> ## Documentation Index
> Fetch the complete documentation index at: https://docs.innate.bot/llms.txt
> Use this file to discover all available pages before exploring further.

# Policy-Defined Skills

Policy-defined skills are neural network policies that run end-to-end—taking sensor input and directly outputting robot actions. Unlike code-defined skills where you write explicit logic, policy-defined skills learn behaviors from demonstration.

This approach works well for manipulation tasks where the variability of real-world scenarios makes explicit programming impractical.

## Two Types of Policies

| Type        | Definition                               | Use Case                          |
| ----------- | ---------------------------------------- | --------------------------------- |
| **Learned** | Neural network trained on demonstrations | Tasks requiring visual adaptation |
| **Replay**  | Recorded action sequence                 | Repeatable, fixed motions         |

## Learned Skills (ACT Policy)

Learned skills use **Action Chunking with Transformers (ACT)**—a neural network architecture that takes camera images and joint positions as input, and outputs action sequences.

### Metadata Structure

```json theme={null}
{
    "name": "pick_socks",
    "type": "learned",
    "guidelines": "Use when you need to pick up socks from the floor",
    "execution": {
        "model_type": "act_policy",
        "checkpoint": "act_policy_step_135000.pth",
        "stats_file": "dataset_stats.pt",
        "action_dim": 10,
        "duration": 45.0,
        "start_pose": [-0.015, -0.399, 1.456, -1.135, -0.023, 0.833]
    }
}
```

You normally never write this file by hand — the training pipeline generates and updates it. See [Dataset format](/training/dataset-format#metadata-file) for how it evolves from skill creation to activation.

### Execution Flow

When a learned skill runs, the BehaviorServer:

1. **Load Policy**: Loads the ACT checkpoint into GPU memory

2. **Move to Start Pose**: Moves the arm to the consistent initial configuration from `metadata.json`

3. **Inference Loop (25Hz)**: Every 40ms:

   * Captures frames from the main camera and wrist camera

   * Reads the current 6-DOF joint state

   * Resizes images to 224×224 and normalizes them

   * Runs a forward pass through the policy

   * Sends the first 6 outputs as joint commands to `/mars/arm/commands`

   * Sends outputs 7–8 as base velocity to `/cmd_vel`

4. **Progress Monitoring**: Terminates early when progress exceeds 95%

5. **End Pose**: Optionally returns to a safe configuration

### Key Characteristics

* **Reactive**: Continuously adjusts based on visual feedback

* **Adaptive**: Handles variation in object position, lighting, orientation

* **Coordinated**: Can move arm and base simultaneously

## Replay Skills

Replay skills play back pre-recorded action sequences. Simpler than learned skills, but deterministic and reliable.

### Metadata Structure

```json theme={null}
{
    "name": "wave",
    "type": "replay",
    "guidelines": "Use when greeting someone or saying hello",
    "execution": {
        "model_type": "replay",
        "replay_file": "episode_0.h5",
        "replay_frequency": 50.0,
        "start_pose": [1.577, -0.6, 1.477, -0.738, 0.0, 0.0],
        "end_pose": [1.577, -0.6, 1.477, -0.738, 0.0, 0.0]
    }
}
```

### Execution Flow

1. **Load Recording**: H5 file containing timestamped actions

2. **Move to Start Pose**: Arm moves to recorded initial position

3. **Playback (50Hz)**: Execute recorded actions in sequence

4. **End Pose**: Return to specified configuration

## Execution Pipeline

Both skill types are executed by the **BehaviorServer**:

```text theme={null}
Innate agent calls skill
       |
       v
SkillsActionServer
       |
       v (physical skill detected)
BehaviorServer.ExecuteBehavior
       |
       +-- Learned --> Load policy, run inference loop
       |
       +-- Replay ---> Load H5 file, playback loop
       |
       v
Robot hardware (/mars/arm/commands, /cmd_vel)
```

## Creating New Skills

### Learned Skill Workflow

The whole pipeline runs from the app — each step has its own guide:

1. **[Collect demonstrations](/training/data-collection)**: Teleoperate the robot through the task with the leader arm (50+ episodes)

2. **[Train the policy](/training/train-act-policy)**: Launch an ACT training run on Innate's cloud GPUs

3. **[Deploy](/training/deploy-trained-skill)**: Download the finished checkpoint from the app — activation writes `metadata.json` and places everything in `~/innate-os/workspace/custom_skills/<name>/` for you

### Replay Skill Workflow

Replay skills are created by **recording the motion once** — no hand-written metadata. You teleoperate the robot through the motion, then save the recording as a replay skill. The robot does the rest:

1. **Record once**: Teleoperate through the motion a single time.

2. **Save as a replay skill**: The robot converts the recording to the replay layout, writes the `type: "replay"` `metadata.json` for you, and drops it into `~/innate-os/workspace/custom_skills/<name>/` — ready to run immediately.

Unlike learned skills, replay skills also capture **head movement**, so the robot reproduces where it was looking during the recording.

## Demonstration Quality

The quality of learned skills depends directly on demonstration quality:

| Good Demonstrations           | Poor Demonstrations         |
| ----------------------------- | --------------------------- |
| Consistent starting positions | Variable starting positions |
| Smooth, deliberate motions    | Hesitant, jerky motions     |
| Varied object positions       | Always same position        |
| Include error recovery        | Only perfect executions     |

## Skill Selection Guide

| Scenario                            | Recommended Type |
| ----------------------------------- | ---------------- |
| Task requires visual adaptation     | **Learned**      |
| Object position varies              | **Learned**      |
| Motion must be identical every time | **Replay**       |
| Simple gesture (wave, point)        | **Replay**       |
| Faster development cycle            | **Replay**       |