Physical AI & Humanoid Robotics: Course Specification

Introduction: The Future of Embodied Intelligence

The Paradigm Shift: From Digital to Physical

For the past decade, artificial intelligence has flourished in digital domains—prediction engines optimizing recommendations, language models generating text, and vision systems analyzing images at superhuman speeds. Yet these systems remain fundamentally abstract. They perceive the world through data feeds and actuate change through abstract outputs. They have no skin, no balance, no sense of gravity.

This is changing. The frontier of AI now extends into physical space. Humanoid robots are no longer science fiction—they are becoming practical tools in factories, hospitals, and homes. But a critical bottleneck remains: the gap between the "digital brain" and the "physical body." A language model can understand natural language, but it cannot translate that understanding into coordinated motor control. A vision system can detect objects, but it cannot grasp them without understanding physics, geometry, and force.

Physical AI is the discipline that bridges this gap. It represents the convergence of three traditionally separate fields:

• Embodied Intelligence: AI systems that must understand and operate within the laws of physics
• Robotics: The hardware and control systems that translate digital decisions into physical motion
• Cognitive Systems: Language models and reasoning engines that transform human intent into robot action

This capstone quarter introduces you to all three domains. By the end, you will have designed, simulated, and deployed a humanoid robot that can understand voice commands, plan paths through environments, recognize objects, and manipulate them—all while respecting the real constraints of physics, computation, and time.

Why Humanoid Robots Matter

Humanoid robots are uniquely positioned to succeed in human-centered environments because they share our physical form. A robotic arm bolted to a factory floor can perform a single repetitive task excellently. But a humanoid robot can open doors, climb stairs, move objects from high shelves to low ones, and work alongside humans without requiring environmental modification.

More importantly, humanoids benefit from an enormous dataset: all the video, motion capture data, and interaction logs generated by humans themselves. The same techniques that train language models on human text can train motor control systems on human movement. This means humanoids have the potential to generalize in ways that task-specific robots cannot.

The Three Pillars of This Course

1. The Robotic Nervous System (ROS 2) – How robots communicate and coordinate. You'll learn the middleware that connects perception (what the robot senses) to cognition (what the robot decides) to control (what the robot does).

2. Digital Twins (Simulation) – How to test everything before deploying to expensive hardware. You'll simulate physics, sensors, and AI models in environments that mirror reality.

3. Vision-Language-Action (VLA) – How to turn natural language commands into motor sequences. A user says "clean the table"—the robot understands intent, plans motion, and executes tasks.

What You'll Build

Your capstone project will be an Autonomous Humanoid—a simulated robot that:

• Receives voice commands through natural language
• Understands spatial and temporal constraints
• Plans collision-free paths through complex environments
• Identifies and localizes objects using computer vision
• Performs manipulation tasks with humanoid hands
• Adapts to unexpected obstacles in real-time

This is not a toy project. The skills you develop—ROS 2 architecture, sensor fusion, physics simulation, and LLM integration—are the same skills used by roboticists at leading labs worldwide.

---

Chapter 1: Foundations of Physical AI and Embodied Intelligence

1.1 What is Physical AI?

Physical AI is artificial intelligence applied to systems that exist in and interact with the physical world. Unlike traditional AI systems that process data purely in software, Physical AI systems must:

• Perceive their environment through sensors (cameras, LiDAR, IMUs, force/torque sensors)
• Reason about physical constraints (objects fall down, not up; energy is conserved; collisions are painful)
• Act by controlling actuators (motors, hydraulics, pneumatics) that produce real-world motion
• Learn from interaction with physical environments, not just simulated data

A language model trained on text has learned patterns in human language. A Physical AI system trained on robot interaction has learned patterns in how the world responds to action.

1.2 Embodied Intelligence: Why a Body Matters

Consider a simple question: what is a ramp?

A language model might answer: "An inclined plane used to facilitate the movement of objects between different heights."

A humanoid robot that has never encountered a ramp must learn something different. It must learn:

• At what angle does friction overcome gravity?
• How does my center of mass need to shift to walk up versus walk down?
• What is the relationship between step height and energy expenditure?
• How do I adjust my gait if the surface is slippery?

This is the difference between knowing about physics and understanding through embodiment. The robot's body is not an afterthought to its AI—it is a fundamental part of how it reasons about the world.

Embodied intelligence rests on a simple insight: the constraints of the body inform the structure of thought. A robot with wheels reasons about movement differently than a robot with legs. A robot with two fingers reasons about grasping differently than a robot with five. This is not a limitation—it is a feature. Evolution has shaped animal nervous systems around their bodies. Humanoids should be shaped by their form too.

1.3 The Robotics Stack: From Sensors to Motors

Every robot is a pipeline. Let's trace what happens when your humanoid robot receives a voice command to "pick up the cup":

Stage 1: Perception

• A microphone captures your voice command
• A RealSense camera (RGB + Depth) scans the scene
• A LiDAR builds a 3D map of the environment
• An IMU (inertial measurement unit) tracks the robot's own balance and orientation

These raw sensor signals are analog noise. They must be fused and cleaned.

Stage 2: Sensor Fusion

• Robot Operating System (ROS 2) aggregates these sensor streams
• VSLAM (Visual Simultaneous Localization and Mapping) algorithms determine "where am I?"
• Object detection models identify the cup in 3D space
• IMU data is fused with motor feedback to estimate the robot's posture

Now the robot has a model of the world: I am here, the cup is there, the floor is solid, the walls are here.

Stage 3: Cognition

• OpenAI Whisper transcribes the voice command to text
• A large language model (GPT-4 or similar) parses natural language intent: "pick up the cup" → [reach, grasp, lift, retract]
• A path planner (Nav2, a ROS 2 module) generates a collision-free trajectory from the robot's current location to the cup's location

Now the robot has a plan: I need to move to location X, then reach my arm to position Y with grasp configuration Z.

Stage 4: Control & Actuation

• Low-level motor controllers receive trajectory commands
• Feedback from joint encoders and force/torque sensors adjusts motion in real-time
• Inverse kinematics translates desired hand positions into joint angles
• The robot moves: walking, reaching, grasping, lifting

Stage 5: Closed-Loop Feedback

• The robot checks: did it succeed? Is the cup still in my gripper?
• If not, it adjusts: re-grasping, recovering from failure, trying again

This is the robotics stack: Sensors → Fusion → Cognition → Planning → Control → Actuation → Feedback.

Each stage is a discipline unto itself. This course teaches you all five.

1.4 From Digital Twins to Real Robots: Sim-to-Real

Robots are expensive. Mistakes are costly. Breaking a $50,000 humanoid during your first attempt at motion planning is not ideal.

This is why simulation is central to robotics. You do not build the robot first and debug in the real world. You build it in software first.

Digital Twins are virtual replicas of robots and their environments. Using tools like Gazebo and NVIDIA Isaac Sim, you can:

• Test motion plans without risk
• Generate synthetic training data for perception models
• Verify physics simulations match reality
• Benchmark different control strategies before deployment

But simulation is not perfect. A simulated cup does not behave exactly like a real cup. Simulated friction is not real friction. Simulated latency is not network latency. The transition from simulation to reality—sim-to-real transfer—is a major challenge in robotics.

We address this by:

1. Making simulations as realistic as possible (high-fidelity physics, photorealistic rendering)
2. Training with simulation but deploying with caution
3. Using real sensor data early, so the robot learns real-world constraints
4. Building robust controllers that work even when reality differs slightly from simulation

1.5 The Humanoid Advantage

Why humanoids? Why not robot arms, drones, or quadrupeds?

Humanoids are uniquely suited to human environments:

• Compatibility: Doors, stairs, handles, and furniture are designed for human proportions. A humanoid can use them unchanged.
• Intuition: Humans intuitively understand humanoid motion. If a humanoid walks like we walk, we trust it. If a six-legged insectoid robot moves like a mantis, we find it unsettling.
• Data Abundance: Decades of motion capture data, video, and human biomechanics research can be adapted to humanoid control.
• Generalization: Because humanoids operate across diverse environments, they must learn general principles—not task-specific hacks.

However, humanoids are also the hardest robots to build. Bipedal balance is an unsolved problem in classical robotics—it is why robots fall and humans rarely do. Hands with dexterity require dozens of motors and sensors, all coordinated precisely. The humanoid is the "final boss" of robotics.

By learning to control humanoids, you will have learned to control almost any robot.

1.6 Sensor Systems: The Robot's Senses

A robot perceives through sensors. The major classes are:

Visual Sensors (Cameras)

• RGB cameras capture color images (what does the world look like?)
• Depth cameras (like RealSense) capture distance (where are objects in 3D space?)
• Structured light: projects IR patterns to measure depth in low-light conditions
• Used for: object detection, visual navigation, manipulation

Range Sensors (LiDAR)

• Light Detection and Ranging: scans a laser beam to build a 3D point cloud of the environment
• Fast, accurate, unaffected by lighting
• Used for: SLAM (mapping), obstacle detection, navigation

Inertial Sensors (IMUs)

• Accelerometers measure proper acceleration (including gravity)
• Gyroscopes measure angular velocity (rotation rate)
• Magnetometers measure magnetic field (compass)
• Together they track: orientation, fall detection, balance
• Used for: gait stability, fall prevention, ground truth for SLAM

Force/Torque Sensors

• Mounted at wrists or joints, measure forces and torques applied
• Used for: detecting contact, adjusting grip force, detecting collisions, learning from physical interaction

Audio Sensors (Microphones)

• Capture voice commands, environmental sounds
• Used for: natural language interfaces, sound localization

For this course, your Physical AI kit includes:

• Intel RealSense D435i: RGB + Depth + IMU (essential for SLAM and object detection)
• Generic USB Microphone: Voice command input
• Optional: LiDAR: More robust navigation in complex environments

1.7 The ROS 2 Ecosystem: The Robot's Nervous System

Imagine building a humanoid robot from scratch. The hands need motor commands. The legs need balance feedback. The eyes need object detection results. The ears need speech recognition. The face needs expression. Every component needs to communicate.

Without a structured system, your code becomes a tangled mess. ROS 2 (Robot Operating System 2) solves this by providing:

Standardized Communication

• Nodes are independent processes (one for perception, one for control, one for planning)
• Topics are named data streams (sensor data flows on /camera/rgb, motor commands on /joint_commands)
• Services are request-response operations (ask for the current robot pose, get an answer)
• Actions are long-running tasks (navigate to location X, report progress)

Modularity & Reusability

• Write a SLAM node once, use it on any robot
• Write a motion planner once, use it with any arm
• The community shares thousands of ROS packages—from navigation to manipulation to visualization

Hardware Abstraction

• Same code works with different cameras, different motors, different platforms
• A control algorithm trained on a simulated robot runs on a real one with minimal changes

Distributed Computing

• Nodes run on different machines (your main PC, your edge AI kit, your real robot)
• Low-latency communication over networks
• Redundancy and fault tolerance

ROS 2 is the lingua franca of robotics. Learning it is learning to think like a roboticist.

1.8 Course Roadmap

This quarter is structured around mastering the robotics stack, layer by layer:

Weeks 1-2: Foundations

• What is Physical AI? Why embodied intelligence?
• Overview of humanoid robotics landscape
• Introduction to sensor systems
• The robotics pipeline conceptually

Weeks 3-5: Perception & Communication (ROS 2)

• ROS 2 architecture: nodes, topics, services, actions
• Writing ROS 2 nodes in Python
• Sensor integration: reading camera streams, LiDAR, IMU
• Intro to URDF (robot description format)

Weeks 6-7: Simulation & Digital Twins

• Gazebo: physics simulation engine
• Visualizing robots in Unity
• Simulating sensors in Gazebo
• Introduction to NVIDIA Isaac Sim (high-fidelity)

Weeks 8-10: AI-Powered Perception & Planning

• NVIDIA Isaac ROS: hardware-accelerated perception
• VSLAM: building maps and localizing
• Object detection and 6D pose estimation
• Nav2: path planning for humanoid bipeds

Weeks 11-12: Humanoid Dynamics & Control

• Kinematics and inverse kinematics
• Bipedal locomotion: walking without falling
• Manipulation: reaching and grasping with humanoid hands
• Natural human-robot interaction

Week 13: Vision-Language-Action (VLA)

• Integrating GPT-4 for intent understanding
• Voice commands with Whisper
• End-to-end system: voice input → robot action
• Capstone: Autonomous humanoid in simulation

1.9 What You'll Learn

By the end of this course, you will be able to:

• Design and describe a humanoid robot using URDF, understanding kinematic chains and degrees of freedom
• Set up a ROS 2 system with multiple nodes communicating via topics, services, and actions
• Simulate complex robots and environments in Gazebo and NVIDIA Isaac Sim
• Build perception pipelines: camera calibration, depth processing, object detection, SLAM
• Implement motion planning for humanoid robots: biped gait generation, obstacle avoidance, manipulation
• Integrate language models into robotic systems for natural human-robot interaction
• Deploy code from high-powered PCs to edge AI kits (Jetson) with resource constraints
• Understand sim-to-real transfer: why simulation differs from reality and how to bridge the gap
• Troubleshoot a complete robotic system end-to-end

These are not toy skills. They are used every day by roboticists at Boston Dynamics, Tesla, Sanctuary AI, and leading research labs worldwide.

1.10 Prerequisites & Assumptions

This course assumes:

• Programming: Comfortable with Python. You will write significant amounts of code.
• Linear Algebra: Comfortable with vectors, matrices, rotations. We will use them frequently.
• Physics: Basic understanding of forces, torques, momentum, kinematics. We will review briefly, but this is not an intro physics course.
• Patience: Robots are complex. Debugging hardware-software integration is frustrating. You will get stuck. That is normal.

If you are rusty on math, we provide review materials in the appendix.

1.11 Hardware & Setup: A Preview

Physical AI requires powerful computational hardware. This is not a course you can run on a laptop.

The Digital Twin Workstation

• NVIDIA RTX GPU (RTX 4070 Ti or better, 12GB+ VRAM)
• 64 GB RAM minimum (32 GB will crash)
• Intel Core i7 (13th Gen+) or AMD Ryzen 9
• Ubuntu 22.04 LTS
• Why: Isaac Sim, physics simulation, and VLA models are computationally intensive

The Physical AI Edge Kit (~$700)

• NVIDIA Jetson Orin Nano (8GB)
• Intel RealSense D435i camera
• ReSpeaker USB microphone
• SD card, power supply
• Why: This is where your code deploys to in real-world conditions

Optional: Physical Robot Hardware

• Unitree Go2 quadruped (~$2,000-$3,000, recommended for budget labs)
• Unitree G1 humanoid (~$16,000, premium option for full capstone deployment)
• Why: Moving from simulation to reality

We will provide detailed setup instructions and troubleshooting guides in Chapter 2.

1.12 How to Use This Course

Each week combines:

• Lecture: Conceptual foundations and big-picture thinking
• Tutorials: Step-by-step guided labs with working code
• Projects: Unguided challenges where you apply concepts
• Readings: Research papers and documentation for deeper understanding

The capstone integrates everything: you will build a complete system from sensors to motor commands, from voice input to manipulation output.

Come ready to code, debug, break things, and fix them. Welcome to Physical AI.