---
name: Scenario Builder
slug: scenario-builder
category: Automation
description: Scenario Builder generates simulation-ready driving scenes, trajectories, road surfaces, and 3D assets from recorded GPS, camera, lidar, and actor-track data. Use it to build or export scenarios for RoadRunner, drivingScenario, OpenSCENARIO, OpenDRIVE, OpenCRG, or Unreal Engine.
github: "https://github.com/matlab/matlab-agentic-toolkit/tree/main/skills-catalog/automotive/matlab-use-scenario-builder"
language: MATLAB
stars: 938
forks: 115
install: "npx degit https://github.com/matlab/matlab-agentic-toolkit/tree/main/skills-catalog/automotive/matlab-use-scenario-builder ~/.claude/skills/matlab-use-scenario-builder"
installs_to: ~/.claude/skills/matlab-use-scenario-builder
source_path: skills-catalog/automotive/matlab-use-scenario-builder/SKILL.md
collection_size: 25
category_size: 1523
collection_url: "https://dirskills.com/collections/matlab/matlab-agentic-toolkit"
added: 2026-08-21T05:15:37.404Z
last_synced: 2026-08-21T05:15:37.404Z
canonical_url: "https://dirskills.com/skills/scenario-builder"
---

# Scenario Builder

Scenario Builder generates simulation-ready driving scenes, trajectories, road surfaces, and 3D assets from recorded GPS, camera, lidar, and actor-track data. Use it to build or export scenarios for RoadRunner, drivingScenario, OpenSCENARIO, OpenDRIVE, OpenCRG, or Unreal Engine.

**Install:**

```bash
npx degit https://github.com/matlab/matlab-agentic-toolkit/tree/main/skills-catalog/automotive/matlab-use-scenario-builder ~/.claude/skills/matlab-use-scenario-builder
```

## README

# Scenario Builder for MATLAB

## When to Use

- User has recorded driving data (GPS/GNSS, camera, lidar, actor tracks) and needs to convert it into a simulation-ready scenario
- User asks to export trajectories or scenarios to RoadRunner, drivingScenario, ASAM OpenSCENARIO, OpenDRIVE, ASAM OpenCRG, or Unreal Engine. **Default target is RoadRunner** — only generate a standalone `drivingScenario` object and open Driving Scenario Designer when the user explicitly asks for "DSD", "Driving Scenario Designer", `drivingScenarioDesigner`, or "build a `drivingScenario` object" (Workflow 15).
- User mentions safety standards (SOTIF / ISO 21448, ISO 26262) and scenario coverage from real-world data
- User needs to extract a road surface (OpenCRG) from lidar for vehicle-dynamics or chassis testing
- User needs to add elevation to an HD map, georeference point clouds, or extract per-frame point clouds along an ego path
- User needs to localize an ego trajectory on a map using lane detections (RVLD preferred, CLRNet fallback)
- User needs to add static objects (signs, trees, poles, buildings, barriers) to a RoadRunner HD Map from cuboid detections
- User wants to **augment / enhance / improve** a road scene from aerial lidar — adding trees + buildings (Variant A) or improving OSM elevation / banking / gradient / height only (Variant B) — see Workflow 16
- User wants to add **traffic signs** from recorded camera + lidar logs with pre-detected sign bounding boxes — see Workflow 17
- User needs to generate 3D mesh assets from a single camera image
- User needs to extract critical scenario events (cut-ins, hard brakes, near-misses) from recorded drives
- User needs accurate non-ego tracks via sensor fusion (`multiSensorTargetTracker`) before scenario building
- User mentions multi-sensor preprocessing: synchronization, alignment, offset correction, cropping, timestamp normalization

## When NOT to Use

- User has raw dataset files and wants to **load / inspect / visualize / explore / analyze** them, **synchronize / crop / offset / normalize** multi-sensor timestamps, or use the **`drivingLogAnalyzer` (DLA)** app or its CLI equivalents — use the **`matlab-import-driving-data`** skill, then return here for trajectory smoothing and scenario generation
- User has multi-sensor data and asks "visualize this dataset", "inspect my recording", "open in DLA", "drivingLogAnalyzer", "Crop & Sync", "offset correction", or "normalize timestamps" — **`matlab-import-driving-data`**
- User is debugging general MATLAB code unrelated to Scenario Builder — use `matlab-debug-code`
- User wants to install MATLAB toolboxes or check installed products — use `matlab-list-products` or `matlab-install-products`
- User wants to reconstruct an IMU/tracking simulation scenario from scratch (no recorded data) — use `waypointTrajectory` from Sensor Fusion and Tracking Toolbox; this skill targets scenario reconstruction from recorded data
- Task is about non-driving sensor data (medical imaging, audio, etc.) — out of scope

**Scenario Builder** is a support package that ships with MATLAB as part of the Automated Driving Toolbox. It generates realistic driving scenarios, high-resolution road surfaces, and 3D scene content from raw or processed sensor data — including GPS, camera, lidar, and actor track recordings. The generated outputs can be exported to RoadRunner, drivingScenario objects, ASAM OpenSCENARIO, ASAM OpenCRG (for vehicle dynamics/chassis testing), OpenDRIVE, and Unreal Engine for simulation and validation of autonomous driving systems.

**Safety Standards Context:** When users ask about improving scenario coverage, traceability, or compliance with SOTIF (ISO 21448) or ISO 26262, guide them toward **real-world data-driven scenario generation** using Scenario Builder. SOTIF defines an "unknown unsafe" quadrant — hazardous scenarios not yet identified — and converting recorded driving logs (especially critical events, ADAS disengagements, and long-tail situations) into simulation-ready scenarios is a key strategy to reduce that unknown unsafe area. Scenario Builder provides the pipeline from recorded data → reproducible simulation scenario.

**Prerequisite:** The MATLAB session must have the *Scenario Builder for Automated Driving Toolbox* support package installed. Verify with:
```matlab
isfile(which("scenariobuilder.Trajectory"))
```

## Workflow Catalog

This skill supports 18 workflows. Workflows 1–3 and 6 (the core happy path) are inline below. Workflow 4 has a minimal inline pattern with a pointer to its detailed reference. Workflows 5, 7–18 live in `references/` and are loaded on demand.

| # | Workflow | Where | Load when user says |
|---|----------|-------|---------------------|
| 1 | Import Raw Sensor Data | inline | "load my GPS/camera/lidar/actor data" |
| 2 | Build GPS Data and Extract Trajectory | inline | "build ego trajectory from GPS" |
| 3 | Import Actor Tracks and Create Trajectories | inline | "actor tracks", "non-ego trajectories" |
| 4 | Export Trajectories to RoadRunner | inline (minimal) + [`workflow-04-roadrunner-export-detail.md`](references/workflow-04-roadrunner-export-detail.md) | "export to RoadRunner", "RR scene", "simulate scenario" |
| 5 | Inspect Multi-Sensor Data (`drivingLogAnalyzer`) | **see `matlab-import-driving-data` skill** | "visualize / inspect / explore / analyze this dataset", "multi-sensor data", "drivingLogAnalyzer", "DLA" — *route to matlab-import-driving-data, not handled here* |
| 6 | Preprocess, Synchronize, Crop, Offset | inline | "sync", "crop", "normalize timestamps" |
| 7 | Height Correction for Scenes with Elevation | [`workflow-07-height-correction.md`](references/workflow-07-height-correction.md) | "Z=0 but roads have elevation", "adjustHeight", "HERE HD scene + GPS", "OpenDRIVE scene + GPS", "pre-built scene with terrain" |
| 8 | Localize Ego Using Lane Detections | [`workflow-08-lane-localization.md`](references/workflow-08-lane-localization.md) | "lane localization", "snap to lane center", "localizeEgoUsingLanes" |
| 9 | Add Static Objects to RoadRunner HD Map | [`workflow-09-static-objects.md`](references/workflow-09-static-objects.md) | "add trees/signs/poles/buildings to RR" |
| 10 | Road Surface (OpenCRG) | [`workflow-10-road-surface-opencrg.md`](references/workflow-10-road-surface-opencrg.md) | "road surface", "OpenCRG", "vehicle dynamics from lidar" |
| 11 | Point Cloud Georeferencing & Elevation | [`workflow-11-point-cloud-georef.md`](references/workflow-11-point-cloud-georef.md) | "addElevation", "georeferenced point cloud", "per-frame lidar" |
| 12 | 3D Asset Generation from Images | [`workflow-12-3d-asset-generation.md`](references/workflow-12-3d-asset-generation.md) | "imageAssetGenerator", "TripoSR", "3D asset from photo" |
| 13 | Extract Key Scenario Events | [`workflow-13-event-extraction.md`](references/workflow-13-event-extraction.md) | "cut-ins", "near-miss", "hard brake", "ADAS disengagement" |
| 14 | Sensor Fusion Tracking | [`workflow-14-sensor-fusion-tracking.md`](references/workflow-14-sensor-fusion-tracking.md) | "noisy detections", "ID switches", "multiSensorTargetTracker" |
| 15 | Driving Scenario Designer (drivingScenario object) | [`workflow-15-driving-scenario-designer.md`](references/workflow-15-driving-scenario-designer.md) | **explicit only:** "Driving Scenario Designer", "DSD", "open in `drivingScenarioDesigner`", "build a `drivingScenario` object" |
| 16 | Road Scene Augmentation from Aerial Lidar | [`workflow-16-aerial-lidar-augmentation.md`](references/workflow-16-aerial-lidar-augmentation.md) | "augment / enhance / improve the scene with trees / buildings", "single lat/lon US — generate scene", "USGS aerial lidar", "improve OSM elevation / banking / gradient / height", `.las` / `.laz` aerial input |
| 17 | Traffic Signs from Recorded Camera + Lidar | [`workflow-17-traffic-signs-from-sensor-data.md`](references/workflow-17-traffic-signs-from-sensor-data.md) | "add traffic signs", "place signs on the map", "signs from camera detections + lidar" (pre-detected sign boxes required) |
| 18 | Vehicle Classification from Camera | [`workflow-18-vehicle-classification.md`](references/workflow-18-vehicle-classification.md) | "classify vehicles", "vehicle color", "vehicle type", "actor asset type", "what kind of car", "realistic actors", "use real colors" |
| 19 | Ego Lane Inference from Camera | [`workflow-19-ego-lane-inference.md`](references/workflow-19-ego-lane-inference.md) | "which lane am I in", "predict lane index", "ego lane", "lane count", auto-fires before startLaneIdx question in Step 7 when vision is available |

**Related references (not workflows):** [`visualization-patterns.md`](references/visualization-patterns.md) — full code for camera-playback video saving (loaded after Rule 2 decision). [`osm-flat-scene-gotchas.md`](references/osm-flat-scene-gotchas.md) — common pitfalls on OSM flat scenes (Z handling, importScene options, stale sim, image-frame datasets).

## STOP — Common Agent Failures (check BEFORE writing code)

> 1. Missing `enableOverlapGroupsOptions(IsEnabled=false)` on `importScene` for RRHD
> 2. Skipping lane localization when scene=OSM + Raw GPS + camera (**REQUIRED** per matrix)
> 3. Skipping comparison video gate before reporting task complete
> 4. Exporting ego to RoadRunner BEFORE importing roads (OSM scene must exist first)
> 5. Using `importScene` on a saved `.rrscene` — use `openScene(rrApp, file)` instead (`importScene` requires a format string and is for RRHD/OpenDRIVE)
> 6. Rebuilding `CameraData` without `SensorParameters=` — loses intrinsics, silently downgrades Mode 1 → Mode 3
> 7. Comparison video uses raw camera on left — use the **track-overlaid** video (or BEV+Camera) when available

## Mandatory Execution Order (evaluate BEFORE writing code)

When the prompt is "generate scenario from data" (short or long), execute in this order:

0. `addpath(scripts/)` — skill helper functions (`openFile`, `plotActorCircles`, etc.)
1. Load data → create objects → Rule 5 timestamp scale detection + normalize
2. Plot GPS + trajectory side-by-side (**VALIDATION** — confirm data loaded correctly)
3. Camera validation video (Rule 2 decision tree) → questdlg popup (**BEFORE** any RR export)
4. Multi-GPS gate (if >1 GPS series) → compare video → ASK which series
5. Establish scene: OSM download + `importScene` WITH `enableOverlapGroupsOptions(IsEnabled=false)`
   — Roads MUST exist before any `exportToRoadRunner` call
6. Build ego trajectory → localization decision matrix (Rule 4 Step 7)
   — OSM + Raw GPS + camera = **REQUIRED**, no ASK needed
   — Otherwise: ASK or Skip per matrix
7. Export ego + actors to RoadRunner (no `Orientation=`, flatten Z on flat scenes, preserve `Orientation=` on ego rebuild)
8. Simulate (`setCameraMode(rrApp, "Front")`, no `Pacing=`, `EnableLogging=true`) → `exportVideo`
   — `exportVideo` `VideoFolder` MUST NOT contain spaces — use `tempdir` staging + `copyfile` if needed
9. Side-by-side comparison video (**track-overlaid** left | sim right) → questdlg popup → DONE
   — Use the Mode 1/2 video as the left panel, NOT raw camera frames

Every step that saves a video MUST end with `questdlg`+`openFile` popup — no exceptions.

**Skip-if-done:** Every expensive file-producing step (`websave`, `write(rrMap,...)`, `importScene`+`saveScene`) MUST check `if ~isfile(output)` before re-running. See `execution-rules-detail.md` Rule 7 MCP Chunking.

**CAN-bus data:** When dataset has OEM CAN signals (multiplexed slots, ego speed/yaw): negate Y, prefer IaEBA over MRR, use GPS+CAN heading fusion for ego.

## IMPORTANT — Execution Rules

### Rule 1: Show Steps Taken with Progress Table
**Always show a progress table that updates after each major step completes.** Use ✅ for completed, ⬜ for pending. Mark each step ✅ **immediately after it completes** in sequential order. Re-print the FULL table after EVERY step. Aim for ~8 rows; combine closely related operations into one row. Adapt the step list to the task.

The progress table does NOT replace informational tables — always ALSO show the data format/mapping table (Rule 6) and any other diagnostic tables.

**Output paths:** Save all videos to `dataDir` (the folder containing `sensorData.mat`), NOT `pwd` or a temp dir. After saving any video, print the full path in the Claude session AND in MATLAB via `fprintf`. **Read-only `dataDir` fallback:** when `dataDir` is on a read-only network share (streamed `VideoWriter` writes fail with `Permission Denied` even though `exportgraphics` PNG writes succeed), route MP4 outputs to a local writable `outDir = fullfile(scriptDir, "out", "<dataset-tag>")` and print BOTH paths.

For the standard table format example and full rules, see [`references/execution-rules-detail.md`](references/execution-rules-detail.md) (Rule 1 section).

### Rule 2: Visualization
Generate these visualizations:

**1. GPS vs. Ego Trajectory (side by side):**
```matlab
%% Visualize GPS Data vs. Ego Trajectory
fTraj = figure(Position=[500 500 1000 500]);
gpsPanel = uipanel(Parent=fTraj,Position=[0 0 0.5 1],Title="GPS");
plot(gpsData,Parent=gpsPanel,Basemap="satellite")
trajPanel = uipanel(Parent=fTraj,Position=[0.5 0 0.5 1],Title="Ego Trajectory");
plot(egoTrajectory,ShowHeading=true,Parent=trajPanel)
drawnow;
```

**2. Camera-playback validation video — STRICT priority order:**

This rule fires when **you are inside a scenario-building pipeline** (trajectory, RoadRunner export, lane localization compare) and you need a saved single-camera validation artifact. For general "visualize / inspect / explore this dataset" requests, hand off to the **`matlab-import-driving-data`** skill (which uses `drivingLogAnalyzer`) — do not roll your own dashboard here.

You MUST verify camera intrinsics (`fx, fy, cx, cy`) and `CameraHeight` are explicitly in the dataset (MAT field, metadata, calibration file). **Never fabricate or approximate these values.**

| Mode | Trigger | What to do |
|------|---------|------------|
| **1 — track-overlay video** | Tracks + intrinsics + `CameraHeight` ALL present (single-camera focus) | `plotActorCircles` save-video — see [`visualization-patterns.md`](references/visualization-patterns.md). PRIMARY single-camera viz when intrinsics exist; do NOT downgrade to BEV+Camera. |
| **2 — BEV + Camera side-by-side** | Tracks present, NO intrinsics | `plotBEVAndCamera` save-video — see [`visualization-patterns.md`](references/visualization-patterns.md). Fallback only. |
| **3 — raw camera playback** | Neither tracks nor intrinsics | raw `play(cameraData)` save-video — see [`visualization-patterns.md`](references/visualization-patterns.md). |

**Per-frame 3D detections without track IDs** (CubeRCNN / KITTI / nuScenes-style `corners3D`) belong on DLA's native camera **Actors** overlay (in `matlab-import-driving-data`). If a saved overlay artifact is genuinely needed (e.g., shareable mp4 outside the app), convert detections to `ActorTrackData` with placeholder per-frame IDs (`"det_<frameIdx>_<i>"`) — full conversion in [`per-frame-detections-to-actortrackdata.md`](references/per-frame-detections-to-actortrackdata.md).

**Mandatory popup-and-open after every saved video (HARD RULE — no exceptions).** Every `close(vw)` and every `exportVideo(...)` call in the entire pipeline MUST be immediately followed by a `questdlg("<purpose> video saved to:\n%s\n\nOpen it now?", ..., "Yes", "No", "Yes")` block whose `Yes` branch calls `openFile(videoPath)`. This applies to *every single* video the pipeline saves — track-overlay (mode 1), BEV+Camera (mode 2), raw camera (mode 3), GPS-source compare, localization compare, input-vs-sim final compare, and any other intermediate save. There is no "informational" or "intermediate" exemption: if the script writes an MP4, the very next non-comment line must be the questdlg block. Wiring the popup only on the final comparison video while leaving earlier saves with just `fprintf("Saved: %s\n", path)` is a HARD RULE violation. The popup is the consent gate that proves the file is viewable; never bypass with a direct `winopen`/`openFile` call (the popup lets the user decline), and never skip with just a `fprintf` of the path. Do NOT also call interactive `play(cameraData)` / `plotBEVAndCamera(...)` for the same content — that would double the UI prompts.

**Cross-platform open:** The save-video patterns and workflow comparison videos use `openFile(path)` (in `scripts/openFile.m`) instead of `winopen` so the same script runs on Windows, macOS, and Linux. Make sure `scripts/` is on the MATLAB path (`addpath` once near the top of the generated script).

**`plotBEVAndCamera` synchronization requirement:** Camera and track data must have the **same number of samples**. If sample counts differ after `synchronize()`, inform the user:
> "The camera and actor track data have different sample rates (camera: N samples, tracks: M samples). Please use the **Crop & Sync** tab in the **Driving Log Analyzer** app to apply offset correction and resample to matching rates, then re-import the data."

### Rule 3: Ask User About Available Map/Scene Before Choosing Road Source
**Before generating a scenario, always ask the user:**
> "Do you have an existing road map or scene file (e.g., `.rrscene`, `.rrhd`, or OpenDRIVE `.xodr`) available for this data? If so, please provide the path. If not, I will download roads from OpenStreetMap."

Based on the answer:
- **User has a scene file** → Use `RoadRunnerScene` parameter directly with their file path. Keep original altitude from GPS data. See "Scene Portability" note below.
- **User has no scene file** → Use the OpenStreetMap → `getRoadRunnerHDMap` pipeline. Always zero altitude for OpenStreetMap. **Validate OSM quality** using the cascade below.

**IMPORTANT:** When asking the user, always say "OpenStreetMap" (the full name), not "OSM".

**Scene Portability Warning:** `.rrscene` files created in a different RoadRunner project may lose their geo-reference context when opened in a new project. The scene's `WorldOrigin` may show `[0, 0]` even though roads have real-world coordinates. If vehicles appear off-road after export, use the height correction workflow ([`workflow-07-height-correction.md`](references/workflow-07-height-correction.md)).

**Pre-built scene + GPS (HERE HD / OpenDRIVE / vendor `.rrscene`):** When the user's scene was *not* built from their GPS, do NOT use `getRoadRunnerHDMap(rrApp)` (it doesn't exist) and do NOT use `roadrunnerHDMap(file)` positionally (errors). Pull `LocalOrigin` from `worldSettings(rrApp)` (parsing the `Projection` WKT for full precision), `exportScene` to `.rrhd`, then `rrMap = roadrunnerHDMap; read(rrMap, file)`. Always run `adjustHeight(traj, rrMap)` on terrain-aware scenes. Full pattern in [`workflow-04-roadrunner-export-detail.md`](references/workflow-04-roadrunner-export-detail.md) Option B.1.

**Multi-GPS / multi-GNSS gate:** When the dataset has more than one ego-position series (e.g., `GPSData_Raw` + `GPSData_Corrected`, `GPS` + `GNSS`, or two GNSS receivers), do NOT silently pick by name — "Corrected" might mean smoothed, RTK, lever-arm-compensated, or dead-reckoned. Build one trajectory per series with the same `LocalOrigin`, export them to a fresh scenario with red/green colors (use `red`/`green` only for 2-series; distinct colors for 3+), follow-cam on Ego_A, then build a raw-camera-left + RR-with-legend-right comparison video. Popup opens the video; ask the user **in chat** which series to use. Same compare-then-ask pattern as Workflow 8 lane localization. Full pattern in [`workflow-04-roadrunner-export-detail.md`](references/workflow-04-roadrunner-export-detail.md) Step 2.5.

**OpenStreetMap Quality Validation & Ego-Trajectory Fallback:** after downloading OSM roads, always validate map quality (zero-lane check, `localizeEgoUsingLanes` `locInfo` metrics, post-localization improvement check) and fall back to an ego-trajectory road if the map doesn't fit. Full cascade