SDK guide / 06

Turn recorded data into evidence without silently rewriting it.

The dataset workflow preserves source order, clock identity, label uncertainty, lineage, and recording manifests so replay and analysis can state exactly what they know.

Dataset workflow

Every transformation leaves an auditable artifact.

01
Ingest
Map source fields and channels explicitly and bind the exact source bytes.
02
Inspect
Inventory episodes, modalities, labels, and timing traces without assigning quality.
03
Synchronize
Assess clock evidence, gaps, coverage, and propagated timing uncertainty.
04
Audit
Review labels, provenance, evidence relationships, and unresolved obligations.
05
Split
Protect lineage families, independence units, duplicates, and real-world holdouts.
06
Replay
Compile one selected failure interval into a simulator-ready evidence capsule.

Optional readers

Install only the formats your workflow uses.

JSONL ingestion is part of the base package. Parquet, MCAP, decoded ROS 2, and split rosbag2 readers are optional dependencies.

terminalShell

pip install "iso-obs[parquet]"
pip install "iso-obs[mcap]"
pip install "iso-obs[mcap-ros2]"  # decoded ROS 2 messages
pip install "iso-obs[rosbag2]"    # split MCAP + metadata.yaml

JSONL and Parquet

Map nested fields and channel clocks explicitly.

A plan names the dataset identity and scope, episode and independence units, timestamp units, channel payload paths, modality IDs, clock IDs, uncertainty, and known limitations.

terminalCLI

iso dataset ingest trajectories.jsonl \
  --plan ingestion-plan.json \
  --output dataset-bundle.json \
  --report ingestion-report.json

iso dataset inspect dataset-bundle.json
iso dataset synchronize dataset-bundle.json --plan synchronization-plan.json
iso dataset audit dataset-bundle.json --plan reliability-audit-plan.json
iso dataset split split-request.json

SOURCE PRESERVATION

Ingestion hashes source bytes and binary payloads. It does not infer labels, clock alignment, scientific split roles, representativeness, or causality.

MCAP and rosbag2

Keep recorder time, publish time, and header time distinct.

MCAP topic rules declare expected schema and message encodings, payload handling, timestamp source, channel clock, and episode windows. Split rosbag2 ingestion additionally verifies metadata.yaml against every ordered MCAP file.

terminalCLI

iso dataset ingest robot-run-42/ \
  --plan rosbag2-ingestion-plan.json \
  --output dataset-bundle.json \
  --report ingestion-report.json \
  --recording-report recording-evidence.json

MANIFESTS ARE CLAIMS

Rosbag2 metadata is verified, not trusted as ground truth. Count, timing, topic, file, and serialization disagreements remain in the recording evidence and route the result to review.

Replay capsules

Separate reconstruction fidelity from label confidence.

Replay compilation joins the immutable bundle, synchronization report, adapter manifest, selected assertions, initial state, random controls, and optional recording evidence.

terminalCLI

iso dataset compile-replay replay-request.json \
  --bundle dataset-bundle.json \
  --synchronization synchronization-report.json \
  --adapter simulation-adapter.json \
  --recording-evidence recording-evidence.json \
  --output failure-replay-capsule.json

TWO INDEPENDENT AXES

A capsule can be exact-replay-ready while its selected failure claim still requires review. Canonical regression construction requires both exact reconstruction and a supported selected claim.

Relevant API

Classes and capabilities across the dataset chain.

Relevant SDK classes and functions
Class or function	Import from	Use it to
`DatasetIngestionPlan`	`iso_obs.dataset_ingestion`	Declare source format, field mappings, timestamps, channels, scope, and limitations.
`DatasetBundle`	`iso_obs.dataset_io`	Carry content-addressed episodes, channels, labels, and timing traces.
`DatasetIngestionReport`	`iso_obs.dataset_ingestion`	Report what was ingested, missing, or routed to review.
`McapDatasetIngestionPlan`	`iso_obs.dataset_mcap`	Declare topic contracts, payload modes, clocks, and episode windows.
`Rosbag2McapIngestionPlan`	`iso_obs.dataset_rosbag2`	Wrap an MCAP plan with split-recording manifest verification.
`Rosbag2RecordingEvidence`	`iso_obs.dataset_rosbag2`	Preserve metadata and ordered file identity plus observed disagreements.
`DatasetSynchronizationPlan`	`iso_obs.dataset_synchronization`	Declare clock evidence, interpolation, gaps, and uncertainty tolerances.
`DatasetReliabilityAuditPlan`	`iso_obs.dataset_reliability`	Audit label status, provenance, episode coverage, and evidence relationships.
`DatasetSplitPlan`	`iso_obs.dataset_splitting`	Protect lineage, independence units, duplicate groups, and real holdouts.
`FailureReplayRequest`	`iso_obs.replay_capsule`	Select episode interval, assertions, channels, initial state, and replay controls.
`FailureReplayCapsule`	`iso_obs.replay_capsule`	Report reconstruction fidelity separately from failure-claim qualification.