Policy-Defined Skills

Policy-defined skills are neural network policies that run end-to-end—taking sensor input and directly outputting robot actions. Unlike code-defined skills where you write explicit logic, policy-defined skills learn behaviors from demonstration. This approach works well for manipulation tasks where the variability of real-world scenarios makes explicit programming impractical.

Two Types of Policies

Type	Definition	Use Case
Learned	Neural network trained on demonstrations	Tasks requiring visual adaptation
Replay	Recorded action sequence	Repeatable, fixed motions

Learned Skills (ACT Policy)

Learned skills use Action Chunking with Transformers (ACT)—a neural network architecture that takes camera images and joint positions as input, and outputs action sequences.

Metadata Structure

{
    "name": "pick_socks",
    "type": "learned",
    "guidelines": "Use when you need to pick up socks from the floor",
    "execution": {
        "model_type": "act_policy",
        "checkpoint": "act_policy_step_135000.pth",
        "stats_file": "dataset_stats.pt",
        "action_dim": 10,
        "duration": 45.0,
        "start_pose": [-0.015, -0.399, 1.456, -1.135, -0.023, 0.833]
    }
}

You normally never write this file by hand — the training pipeline generates and updates it. See Dataset format for how it evolves from skill creation to activation.

Execution Flow

When a learned skill runs, the BehaviorServer:

Load Policy: Loads the ACT checkpoint into GPU memory
Move to Start Pose: Moves the arm to the consistent initial configuration from metadata.json
Inference Loop (25Hz): Every 40ms:
- Captures frames from the main camera and wrist camera
- Reads the current 6-DOF joint state
- Resizes images to 224×224 and normalizes them
- Runs a forward pass through the policy
- Sends the first 6 outputs as joint commands to /mars/arm/commands
- Sends outputs 7–8 as base velocity to /cmd_vel
Progress Monitoring: Terminates early when progress exceeds 95%
End Pose: Optionally returns to a safe configuration

Key Characteristics

Reactive: Continuously adjusts based on visual feedback
Adaptive: Handles variation in object position, lighting, orientation
Coordinated: Can move arm and base simultaneously

Replay Skills

Replay skills play back pre-recorded action sequences. Simpler than learned skills, but deterministic and reliable.

Metadata Structure

{
    "name": "wave",
    "type": "replay",
    "guidelines": "Use when greeting someone or saying hello",
    "execution": {
        "model_type": "replay",
        "replay_file": "episode_0.h5",
        "replay_frequency": 50.0,
        "start_pose": [1.577, -0.6, 1.477, -0.738, 0.0, 0.0],
        "end_pose": [1.577, -0.6, 1.477, -0.738, 0.0, 0.0]
    }
}

Execution Flow

Load Recording: H5 file containing timestamped actions
Move to Start Pose: Arm moves to recorded initial position
Playback (50Hz): Execute recorded actions in sequence
End Pose: Return to specified configuration

Execution Pipeline

Both skill types are executed by the BehaviorServer:

Innate agent calls skill
       |
       v
SkillsActionServer
       |
       v (physical skill detected)
BehaviorServer.ExecuteBehavior
       |
       +-- Learned --> Load policy, run inference loop
       |
       +-- Replay ---> Load H5 file, playback loop
       |
       v
Robot hardware (/mars/arm/commands, /cmd_vel)

Creating New Skills

Learned Skill Workflow

The whole pipeline runs from the app — each step has its own guide:

Collect demonstrations: Teleoperate the robot through the task with the leader arm (50+ episodes)
Train the policy: Launch an ACT training run on Innate’s cloud GPUs
Deploy: Download the finished checkpoint from the app — activation writes metadata.json and places everything in ~/innate-os/workspace/custom_skills/<name>/ for you

Replay Skill Workflow

Replay skills are created by recording the motion once — no hand-written metadata. You teleoperate the robot through the motion, then save the recording as a replay skill. The robot does the rest:

Record once: Teleoperate through the motion a single time.
Save as a replay skill: The robot converts the recording to the replay layout, writes the type: "replay" metadata.json for you, and drops it into ~/innate-os/workspace/custom_skills/<name>/ — ready to run immediately.

Unlike learned skills, replay skills also capture head movement, so the robot reproduces where it was looking during the recording.

Demonstration Quality

The quality of learned skills depends directly on demonstration quality:

Good Demonstrations	Poor Demonstrations
Consistent starting positions	Variable starting positions
Smooth, deliberate motions	Hesitant, jerky motions
Varied object positions	Always same position
Include error recovery	Only perfect executions

Skill Selection Guide

Scenario	Recommended Type
Task requires visual adaptation	Learned
Object position varies	Learned
Motion must be identical every time	Replay
Simple gesture (wave, point)	Replay
Faster development cycle	Replay

​Two Types of Policies

​Learned Skills (ACT Policy)

​Metadata Structure

​Execution Flow

​Key Characteristics

​Replay Skills

​Metadata Structure

​Execution Flow

​Execution Pipeline

​Creating New Skills

​Learned Skill Workflow

​Replay Skill Workflow

​Demonstration Quality

​Skill Selection Guide

Two Types of Policies

Learned Skills (ACT Policy)

Metadata Structure

Execution Flow

Key Characteristics

Replay Skills

Metadata Structure

Execution Flow

Execution Pipeline

Creating New Skills

Learned Skill Workflow

Replay Skill Workflow

Demonstration Quality

Skill Selection Guide