What is Issac Sim?

NVIDIA Isaac Sim™ :
Extensible robotics simulator that gives you a faster, better way to design, test, and train AI-based robots.
It’s powered by Omniverse™ to deliver scalable, photorealistic, and physically accurate virtual environments for building high-fidelity simulations.
Footnote
Extensible Robotics Simulator: This means that Isaac Sim is not just a fixed software package; it can be expanded or customized with additional modules or functionalities. This extensibility is crucial for adapting the simulator to the specific needs of various robotics projects.
Faster and Better Design and Testing: With Isaac Sim, the process of creating and experimenting with robot designs is accelerated. You can quickly iterate through design prototypes and test scenarios, which would be time-consuming, costly, or impractical to perform in the real world.
AI-Based Robots Training: Isaac Sim provides an environment for training the artificial intelligence (AI) systems of robots. This is particularly important for developing and refining the algorithms that enable robots to perceive their environment, make decisions, and perform tasks autonomously.
Powered by Omniverse™: Omniverse is NVIDIA’s platform for creating and operating virtual worlds, and Isaac Sim leverages this technology. This integration means that Isaac Sim can create highly detailed and realistic simulations.
Scalable, Photorealistic, and Physically Accurate Virtual Environments: The simulator can generate virtual environments that are not only visually realistic (photorealistic) but also adhere to the laws of physics (physically accurate). This aspect is crucial for ensuring that the simulations are as close to real-world conditions as possible.
Building High-Fidelity Simulations: High fidelity in simulations refers to the accuracy and detail with which these simulations replicate real-world conditions. Isaac Sim’s ability to create such high-fidelity simulations is essential for effectively testing and training AI-based robots, ensuring that they can operate effectively in real-world scenarios.
-With Isaac Sim, developers and researchers around the world can train and optimize AI robots for a wide variety of tasks
Ex.
(1)Fraunhofer, technological leader in logistics
Uses Issac Sim for the virtual development of Obelix, a highly dynamic indoor/outdoor autonomous mobile robot(AMR).
-After importing over 5400 parts from CAD and rigging with Omniverse PhysX, virtual robot moves as deftly in simulation as it does in the real world
-Accelerates virtual development but also enables scaling to larger scenarios
Footnote
Importing and Rigging Process
Importing Over 5400 Parts from CAD: Computer-Aided Design (CAD) software is used to design the parts of Obelix. These designs, numbering over 5400 individual components, are then imported into Isaac Sim.
Rigging with Omniverse PhysX: Once imported, these parts are rigged using Omniverse PhysX, which is a physics engine developed by NVIDIA.
Rigging involves setting up the virtual parts so they can move and interact realistically in the simulation, mimicking real-world physics.
Simulation Capabilities
Realistic Movement in Simulation: The virtual model of Obelix in Isaac Sim is able to move as adeptly as it would in the real world. This means the simulation accurately reflects the robot’s physical capabilities and how it would navigate and operate in real-life scenarios.
Acceleration of Virtual Development: Using this simulation significantly speeds up the development process. Problems can be identified and solved virtually, which is faster and less costly than making changes to a physical prototype.
Scaling to Larger Scenarios: The simulation also allows for testing in various, potentially larger, scenarios than what might be feasible with a physical prototype. This scalability is crucial for ensuring that Obelix can operate effectively in a wide range of environments and conditions.
AMR Technologies: AMR stands for Autonomous Mobile Robots. These robots are designed to move around and perform tasks without direct human control. They are commonly used in various industries for tasks like transportation, inspection, and inventory management.
Advancements in AMR Technologies
Fast Moving AMRs: Fraunhofer is working on advancing AMR technologies to develop robots that move much faster than the current models available in the market.
Real-World Application Simulations: Before building these advanced AMRs, Fraunhofer uses simulations to mimic real-world conditions. These simulations help identify potential issues early in the development process, saving time and resources that would otherwise be spent on building and testing physical models.
Use of Simulations
Cost-Effectiveness: By simulating various scenarios, they can foresee and address problems without the high costs associated with physical prototyping and testing.
Real-World Testing: These simulations are not just theoretical but are designed to reflect actual conditions and challenges the AMRs would face in real environments.
Development of the O3dyn(Obelix)
O3dyn: This is a specific type of fast-moving AMR currently being developed.
NVIDIA Jetson Edge AI and Robotics Platform: The O3dyn uses this platform as its control unit. The NVIDIA Jetson is a series of embedded computing boards designed for AI and robotics applications. It’s powerful enough to process data from various sensors and control the robot effectively.
Capabilities of the O3dyn:
Speed: It can move at speeds over 30 mph, which is significantly fast for an AMR.
Maneuverability: Thanks to AI assistance, it can navigate tight spaces. This ability is crucial for AMRs that need to operate in crowded or complex environments.
Control of Electronics and Sensors: The control unit manages all the electronic components and sensors necessary for the safe operation of the O3dyn, ensuring it can respond appropriately to its environment.
(2)Festo, well known for industrial automation
-Uses Issac Sim to develop intelligent skills for collaborative robots(cobots) requiring acute awareness of their environments, human partners, and tasks
-Festo uses cortex, an Issac Sim tool that dramatically simplifies programming cobot skills
-For perception, Ai models used in this tasks were trained only using synthetic data generated by Issac replicator
Footnote
Festo: Known for its expertise in industrial automation, Festo is a leading company in the field of automation technology. Development of Cobots
Collaborative Robots (Cobots): These are robots intended to physically interact with humans in a shared workspace. Unlike traditional industrial robots, cobots are designed to work alongside human workers, assisting them rather than replacing them.
Intelligent Skills Development: Festo uses Isaac Sim to develop advanced skills for these cobots. These skills include acute awareness of their environments, human partners, and the specific tasks they are assigned to perform. This awareness is crucial for ensuring safe and efficient interaction between humans and robots.
Festo’s Use of Cortex
Cortex in Isaac Sim: Cortex is a tool within Isaac Sim. It significantly simplifies the programming of cobot skills. In other words, Cortex makes it easier for developers to equip cobots with the necessary capabilities for their intended functions. Simplifying Programming: The complexity of programming cobots is reduced with Cortex, allowing for more efficient development and implementation of essential skills.
AI and Perception
AI Models and Perception: For a cobot to interact safely and effectively with humans and its environment, it must have a sophisticated perception system. This system is powered by AI models.
Training with Synthetic Data: Interestingly, the AI models used by Festo for these tasks were trained exclusively with synthetic data generated by Isaac Replicator. Isaac Replicator is another tool within the Isaac Sim ecosystem that allows for the creation of realistic, synthetic datasets.
Benefits of Synthetic Data: Using synthetic data for training has several advantages. It allows for the creation of diverse and challenging scenarios which might be difficult or expensive to capture in the real world. It also ensures that the AI models can generalize well across various real-world situations.
(3)Anymal, robot dog developed by a leading robotics research group from Eth Zurich and Swiss-Mile
-Using end to end GPU accelerated by reinforcement learning, Anymal, whose wheels were replaced with its wheels, learned to walk over urban terrain within minutes, rather than weeks using Nvidia’s Issac gym training tool
-Locomotion policy was verified in Issac Sim and deployed on real Anymal
-This is a compelling demonstration of simulator training for real-world-deployment
-From training perception and policies to hardware-in-loop, Issac Sim is the tool to build Ai-based robots that are born in simulation to work (and play) in the real world
Footnote
Developed by ETH Zurich and Swiss-Mile: Anymal is a robotic dog developed by a leading robotics research group from ETH Zurich in collaboration with Swiss-Mile. ETH Zurich is renowned for its expertise in robotics and engineering. Advanced Training Using GPU Acceleration and Reinforcement Learning
End-to-End GPU Acceleration: Anymal’s training process involves end-to-end GPU (Graphics Processing Unit) acceleration. This means that every step of the training process, from data processing to learning algorithms, is accelerated using powerful GPU computing, making the process much faster and more efficient.
Use of NVIDIA’s Isaac Gym Training Tool: Anymal was trained using Isaac Gym, a tool developed by NVIDIA for training AI in simulated environments. This tool is specialized for robotic applications, allowing for rapid training and testing of different behaviors.
Learning to Walk Over Urban Terrain
Wheels Replaced with Legs: An interesting aspect of Anymal’s development is that it initially had wheels, which were later replaced with legs. This transition is significant in robotics as walking with legs is a much more complex and dynamic process than rolling on wheels.
Rapid Learning Process: Using the mentioned techniques and tools, Anymal learned to walk over urban terrain within minutes - a task that would traditionally take weeks. This accelerated learning process is a testament to the efficiency of GPU-accelerated reinforcement learning.
Verification and Deployment
Locomotion Policy Verification in Isaac Sim: After the initial training, Anymal’s walking or locomotion policy was further verified in Isaac Sim. This simulation ensured that the learned behaviors were safe and effective.
Deployment on Real Anymal: Once verified, the policy was deployed on the actual Anymal robot, enabling it to move and operate in the real world.
Implications for Real-World Deployment
Demonstration of Simulator Training Efficacy: This development process showcases how training in a simulator can be effectively transferred to real-world applications, a crucial aspect of modern robotics.
From Training to Real-World Application: Isaac Sim proves to be an invaluable tool in this context. It not only allows for training perception and decision-making policies but also facilitates hardware-in-the-loop simulations. This means the virtual training environment can incorporate and adapt to the physical characteristics of the actual robot hardware.

Release Notes

https://docs.omniverse.nvidia.com/isaacsim/latest/release_notes.html

Installation

Issac Sim Requirements

System Requirements
=>Fulfilled
Issac Sim Driver Requirements
=>Fulfilled

Workstation Installation

Workstation Setup
(1) Download Omniverse Launcher
(1)-1 Download Omniverse Launcher
Link: https://www.nvidia.com/en-in/omniverse/download/
Select Standard => Download Free
(1)-2 Search for the file omniverse-launcher-linux.AppImage in your pc file navigator(Usually Downloaded in the Downloads folder)

(1)-3 Right click mouse -> Select “properties”
Select Permissions tab -> Check on Allow executing file as as program -> Exit

(1)-4 First Run
Double click the AppImage

Omniverse-launcher pops out & Agree on license agreement
(1)-5 Path
Installation Path confirmation

Check path for installation
(1)-6 Install cache


(2)Install Cache from the Omniverse Launcher
Downloaded previously

(3)Install Nucleus from the Omniverse Launcher
(3)-1 Click on Nucleus Tab

(3)-2 Click on Create Local Server

Set preferred data path and click Next
(3)-3 Create Administrator account

(4)Install Visual Studio Code to view and debug source code
(4)-1 Search for Visual studio code in Ubuntu Software

(4)-2 Click on Install

Omniverse Launcher
(1)Click on Exchange Tab and Issac Sim can be found in the apps corner

(2)Click on Install

(3)Click Library and select Isaac Sim to Launch

Isaac Sim App Selector

-Mini-windowed app that helps run Isaac Sim in different modes
-Isaac Sim App Selector will run as default when launching Isaac Sim from the Omniverse Launcher
Click on Start!

Isaac Sim First Run

Container Installation

Container Setup
(1)Install NVIDIA Driver
=>Already Installed
(2)Install Docker
=>Docker 24.0.7 already Installed
(3)Install NVIDIA Container Toolkit
Using Omniverse Streaming Client
(1)Installation and Usage
From the Omniverse Launcher, select Exchange from the top-level menu bar

Click Omniverse Streaming Client from the left-hand Apps sidebar, click the Install button

Reference:
https://docs.omniverse.nvidia.com/streaming-client/latest/user-manual.html
(2)Configuration

Python Environment Installation
Possible to run Isaac Sim natively from Python rather than as a standalone executable. This provides more low-level control over how to initialize, setup, and manage an Omniverse application.
(1)Default Python Environment
Isaac Sim provides a built-in Python 3.10 environment that packages can use, similar to a system-level Python install. We recommend using this Python environment when running the Python scripts
Start a python script

./python.sh path/to/script.py
(2)Jupyter Notebook Setup
Jupyter Notebooks that use Isaac Sim can be executed as follows
./jupyter_notebook.sh path/to/notebook.ipynb
(3)Visual Studio Code Support
The Isaac Sim package provides a .vscode workspace with a pre-configured environment
Launch configurations for running in standalone Python mode, or the interactive GUI
An environment for Python auto-complete
You can open this workspace by opening the main Isaac Sim package folder in Visual Studio Code (VSCode)
Open this workspace by opening the main Isaac Sim package folder in Visual Studio Code (VSCode)
(3)Advanced: Running In Docker
Start the Docker container following the instructions in Container Deployment up to step 8.
Once the Isaac Sim container is running, you can run a Python script or Jupiter Notebook from the sections above.
(4)Advanced: Running with Anaconda
Create a new environment with the following command
If you have an existing conda environment, ensure that the packages in environment.yml are installed. Alternatively, you can delete and re-create your conda environment
Finally, you need to set up environment variables so that Isaac Sim python packages are located correctly.
You can then run samples as follows in the isaac-sim conda env

ROS & ROS 2 Installation
Omniverse Isaac Sim provides both a ROS and ROS 2 bridge for ROS system integration. The same set of common components are used to define the types of data being published/received by the simulator
Platform: Ubuntu 20.04
Ros: Noetic
Ros2: Foxy
For the ROS bridge, Isaac Sim runs a custom roscpp build of ROS Noetic internally so that it works properly with the Omniverse framework and Python 3
For the ROS 2 bridge, Isaac Sim is currently compatible with ROS 2 Foxy and ROS 2 Humble
Running Native ROS
-Method of ROS 2 installation determines which features of ROS 2 can be used. Isaac Sim comes with Python3.10, if you have installed ROS 2 with a different version of Python a fallback rclpy will be used which comes pre-packaged with Isaac Sim and compiled with Python3.10
-rclpy: Using ROS 2 functionality with Python in Isaac Sim
The ROS_DISTRO env variable is used to determine if ROS 2 is sourced and which distro to use, if this variable is not set an internal ROS 2 distro build will be used. Message definitions can be different between ROS 2 versions, due to this, we dynamically load the appropriate ROS 2 backend depending on which ROS Distro is sourced.
(1)Download ROS 2 following the instructions on the official website
https://docs.ros.org/en/foxy/Installation/Ubuntu-Install-Debians.html
(1)-1 Resources
ROS 2 Foxy (Ubuntu Focal): amd64, arm64
(1)-2 System setup
Set locale




Make sure you have a locale which supports UTF-8
(1)-3 Setup Sources

Ensure that the Ubuntu Universe repository is enabled


Add the ROS 2 GPG key with apt

Add the repository to your sources list.
(1)-4 Install ROS 2 packages


Update your apt repository caches after setting up the repositories
ROS 2 packages are built on frequently updated Ubuntu systems
Ensure system is up to date before installing new packages

Desktop Install (Recommended): ROS, RViz, demos, tutorials

ROS-Base Install (Bare Bones): Communication libraries, message packages, command line tools. No GUI tools

Development tools: Compilers and other tools to build ROS packages
(1)-5 Environment setup
Sourcing the setup script

Set up your environment by sourcing the following file
Try Some examples


(1)-6 Avoid sourcing bash script every time
Problem: Annoying to always source a script every time we open terminal
Solution: Open home directory and so you can open the bashrc


(1)-7 Check whether it is scripted properly in ros2 script

Ros2 installation complete & Environment sourced properly
(2)Install vision_msgs_package

(3)Ensure that the ROS environment is sourced in the terminal or in your ~/.bashrc file. You must perform this step each time before using any ROS commands and running Isaac Sim
=>Already Done

Enabling the ROS Bridge Extension
If you intend to use a ROS2 bridge, before launching Isaac Sim, you need to set the Fast DDS middleware on all terminals that will be passing ROS2 messages
If you are not using a Isaac Sim ROS2 Workspace, create a file named fastdds.xml under ~/.ros/, paste the following snippet link into the file
Create the fastdds.xml File

Set the FASTRTPS_DEFAULT_PROFILES_FILE Environment Variable

Set ROS_DOMAIN_ID (Optional)

Source Your ROS 2 Installation and Workspace

Launch Isaac Sim

Launch ROS 2 Nodes

Setting Up Workspaces
(1)Clone the Isaac Sim ROS Workspace Repository from https://github.com/NVIDIA-Omniverse/IsaacSim-ros_workspaces
(2)A few ROS packages are needed to go through the Isaac Sim ROS / ROS 2 tutorial series. To make it easy, entire ROS and ROS2 workspaces with the necessary packages are included. => Ubuntu > Foxy
(2)-1 To build the ros2 workspace, you may need to install additional packages

For rosdep install command

For colcon build command
(2)-2 Ensure your native ROS2 has been sourced if not already

(2)-3 Resolve any package dependencies from the root of the ROS2 workspace by running the following command



(2)-4 Build the workspace
Error

Solution: (Searching for it…)
https://forums.developer.nvidia.com/t/troubleshooting-broken-ros-packages-in-isaac-sim-workspace-setup/278881
=> Couldn’t find a solution for building it from source

Way to diverge the Error:
Use the IsaacSim ROS Workspace Repository (The one cloned in the Isaac Sim ROS Workspace Repository from https://github.com/NVIDIA-Omniverse/IsaacSim-ros_workspaces)

Under the root directory, new build, install, and log directories will be created

(2)-5 Open a new terminal and source the workspace with the following commands

Add a line in bashrc, that sources the IsaacSim ROS Workspace



Building ROS 2 Workspaces with Python3.10
If you want to use rclpy with Isaac Sim your ROS 2 workspace must be built with Python3.10
Dockerfiles which builds minimal dependencies of ROS 2 with Python3.10 is provided
Ubuntu20 > Foxy
Minimal dockerfile will be used to build our workspace with Python3.10
(Ensure that docker is installed on your system) -> Installed
(1)Clone Isaac Sim ROS Workspace
=> Already Done
(2)Build dockerfile

Minimal foxy_ws needed to run Isaac Sim is under build_ws/foxy/foxy_ws
Additional workspaces can also be created and built in this dockerfile

(3)Now source the ROS 2 workspace
Run Isaac Sim from the same terminal, the sourced workspace contains the minimal ROS 2 Foxy dependencies needed to enable the ROS 2 bridge

Reference:
https://velog.io/@raise_wise/2.-Omniverse-Isaac-Install
https://docs.omniverse.nvidia.com/launcher/latest/installing_launcher.html

https://docs.omniverse.nvidia.com/install-guide/latest/workstation-install.html