Topic 4 — Unity & High-Fidelity Visualization

Gazebo and Isaac Sim cover most of your simulation needs, but Unity is a powerful additional tool for building interactive, visually polished demos and experiments. This topic introduces Unity as a robotics platform, explains when to use it, and shows how to integrate it with ROS 2 as part of a multi-tool workflow alongside Gazebo and RViz.

---

4.1 Why Unity for Robotics?

Unity is a widely used game engine with:

• A mature physics engine (PhysX, similar lineage to Isaac Sim).
• A vast asset ecosystem (3D models, environments, characters).
• A rich toolchain for UI, interaction, and animation.

Unity is particularly useful for:

• High-fidelity visualization for demos, user studies, and presentations.
• Interactive debugging: moving robots and objects via GUI, testing edge cases.
• Rapid environment prototyping with off-the-shelf assets.

Compared to Gazebo and Isaac Sim:

• Gazebo: Robotics-first, strong ROS integration, modest visuals.
• Isaac Sim: Robotics + photorealistic data generation, GPU-heavy.
• Unity: General-purpose engine, excellent visuals and interaction, lighter robotics ecosystem (but improving).

---

4.2 Unity Setup & ROS 2 Bridge (Conceptual)

To use Unity with your ROS 2 system, you typically:

1. Install Unity 2022+ and create a 3D URP (Universal Render Pipeline) or HDRP project.
2. Add a ROS–Unity bridge package (e.g., ROS–TCP Connector or equivalent).
3. Configure ROS connection parameters:
   • ROS 2 domain ID.
   • IP/port for bridge node.
4. Create C# scripts that:
   • Subscribe to ROS 2 topics (joint states, robot pose, sensor data).
   • Publish commands or events back to ROS 2 nodes (e.g., teleoperation input).

From ROS 2’s perspective, Unity is “just another node” on the network:

• It subscribes to joint_states and visualizes motion.
• It may subscribe to /tf or pose topics to render the robot and environment.
• It can publish user commands (e.g., target poses or waypoints).

---

4.3 Robot Visualization in Unity

Importing Robot Models

Common workflow:

1. Convert URDF meshes to a Unity-friendly format (.fbx, .obj).
2. Import meshes into Unity as GameObjects.
3. Rebuild the joint hierarchy:
   • One GameObject per link.
   • Parent/child relationships match URDF.
4. Add colliders and rigidbodies as needed for physics interaction.

Applying Joint Angles

A simple C# script can map ROS 2 joint angles to Unity transforms:

• Subscribe to a sensor_msgs/JointState equivalent.
• Maintain a dictionary from joint name → GameObject transform.
• For each update:
  • Convert joint angle to rotation (e.g., around local X/Y/Z).
  • Apply to the corresponding transform.

This allows Unity to mirror the pose of the simulated or real humanoid robot.

---

4.4 Lighting, Materials & Performance

Unity shines (literally) in its handling of lighting and materials:

• Use physically based rendering (PBR) materials for realistic surfaces.
• Configure direct lights, indirect lighting, and shadows.
• Adjust post-processing (bloom, color grading) for polished visuals.

For performance:

• Use Level of Detail (LOD) groups for far-away objects.
• Combine static meshes (batching) where possible.
• Limit expensive real-time shadows and reflection probes.

The goal is not perfect film-level rendering, but clear, compelling visualization that runs in real time (~60 FPS) during demos or interactive debugging.

---

4.5 Multi-Tool Workflow: Gazebo, RViz, and Unity

Each tool has a distinct role:

• Gazebo: Primary physics simulation and automated testing.
• RViz: Low-level debugging and introspection of ROS 2 topics and frames.
• Unity: High-level visualization and user-facing interaction.

Example workflow for a humanoid navigation task:

1. Plan & Simulate in Gazebo:
   • Run physics simulation with realistic friction and mass.
   • Validate controllers and planners for stability and collision avoidance.
2. Debug with RViz:
   • Visualize maps, sensor data, and trajectories.
   • Inspect TF frames and topic connectivity.
3. Visualize in Unity:
   • Stream joint states and robot pose into Unity.
   • Render an appealing environment with lighting, textures, and UI overlays.
   • Record videos or run interactive demos.

Synchronization considerations:

• Use consistent frame names and conventions across tools.
• If necessary, designate a single “source of truth” for robot pose (e.g., map or world frame from ROS 2).
• Ensure time bases are aligned (ROS simulation time vs. Unity time).

---

4.6 Hands-On Exercise: Unity Visualization of Your Humanoid

This exercise (lighter than a full lab) introduces Unity into your workflow.

Tasks

1. Create a simple Unity scene:
   • Floor, a few walls, and basic lighting.
   • Import a simplified humanoid mesh or proxy model.
2. Set up ROS 2 bridge:
   • Configure connection to your ROS 2 graph.
   • Subscribe to joint_states and a base pose topic (e.g., /humanoid/base_pose).
3. Animate the robot:
   • Write a C# script that:
     • Maps joint names to GameObjects.
     • Applies joint angles each frame.
   • Optionally, visualize the robot’s path (trail renderer or line).
4. Use Gazebo as the “source of truth”:
   • Run your humanoid simulation in Gazebo.
   • Ensure ROS 2 publishes joint states and base pose.
   • Unity simply visualizes those states, frame by frame.

Outcome

• Gazebo handles physics and sensors.
• ROS 2 handles communication and control.
• Unity renders a polished view of your humanoid moving in its environment.

You now have a three-tool stack—Gazebo, RViz, and Unity—that you can mix and match depending on whether you are debugging algorithms or demonstrating capabilities.