SDK guide / 05

Compile reliability questions into reproducible simulation work.

A campaign begins with a regression definition and an adapter contract. Compatibility is checked before compute is spent, and no gate is calculated until the planned work is complete.

Campaign lifecycle

Definition before execution. Evidence after completeness.

01
Define cases
Represent the canonical failure, its neighborhood, its boundary, and positive and negative controls.
02
Declare seeds
Choose the randomization strategy and bind its random stream when applicable.
03
Describe the adapter
State backend identity, simulator evidence, supported controls, and maximum parallelism.
04
Compile
Preflight compatibility and expand the case-by-seed matrix into deterministic work items.
05
Execute
Return an identity-bound result for every work item.
06
Evaluate
Calculate gates only after the full campaign is complete and valid for confirmatory use.

Adapter contract

Declare what the simulator can actually control.

Physics simulators, learned world models, hybrid twins, replay systems, and hardware-in-the-loop backends use the same manifest shape while exposing different capabilities and limitations.

adapter.pyPython

from iso_obs.simulation import (
    BackendType,
    SimulationAdapterManifest,
    SimulationCapability,
)

adapter = SimulationAdapterManifest(
    adapter_name="warehouse-isaac",
    adapter_version="1.2.0",
    backend_type=BackendType.PHYSICS_SIMULATOR,
    simulator_name="Isaac Sim",
    simulator_version="5",
    simulation_evidence_manifest_digest=simulation.content_digest(),
    capabilities=(
        SimulationCapability.SEEDED_RESET,
        SimulationCapability.PARAMETER_OVERRIDE,
        SimulationCapability.ARTIFACT_LOADING,
        SimulationCapability.RANDOM_STREAM_CONTROL,
        SimulationCapability.INVARIANT_SIGNAL_EXPORT,
    ),
    supported_parameters=("floor_friction", "observation_latency_ms"),
    supported_artifacts=("scenario",),
    max_parallelism=8,
    limitations=("Contact severity is not validated.",),
)

Studio rendering

Give Studio declarations it can render without inventing state.

Rendering is driven by versioned data, not screenshots or generated scenery. An environment map needs simulator-declared geometry. A scenario explanation needs typed, versioned perturbations. The two contracts are independent and remain pinned to their respective environment and scenario versions.

01 · ENVIRONMENT

Declare the space.

Provide a coordinate frame, units, source digest, and at least one polygon with three or more points.

Accepted kinds: world · navigable · obstacle · restricted · spawn · goal
Studio result: Labeled top-down map, dimensions, semantic legend, and provenance.
Plan: Pro, Team, or contracted Enterprise

02 · SCENARIO

Declare the controlled change.

Required fields: family · name · parameters
Studio result: Baseline → controlled change → evidence-to-inspect explanation.
Plan: Team or Enterprise

Manifest presentStudio renders only the declared geometry and labels.

Perturbations presentStudio explains the intervention without predicting its outcome.

Input missingStudio shows metadata and an unavailable state; it does not fabricate a view.

1. Register truthful environment geometry

Export polygons from the simulator or environment source of truth. The source_digest must identify that source artifact so the visual can be traced back to the exact declared geometry.

environment-render.jsonJSON

{
  "schema_version": "iso-obs.environment-render.v1",
  "coordinate_frame": "warehouse/map",
  "units": "m",
  "source_digest": "sha256:<simulator-artifact-digest>",
  "boundaries": [
    {
      "id": "world",
      "kind": "world",
      "label": "Warehouse boundary",
      "points": [{"x": 0, "y": 0}, {"x": 24, "y": 0}, {"x": 24, "y": 16}, {"x": 0, "y": 16}]
    },
    {
      "id": "aisle-a",
      "kind": "obstacle",
      "label": "Storage aisle A",
      "points": [{"x": 5, "y": 3}, {"x": 8, "y": 3}, {"x": 8, "y": 13}, {"x": 5, "y": 13}]
    },
    {
      "id": "robot-start",
      "kind": "spawn",
      "label": "Robot start",
      "points": [{"x": 1, "y": 1}, {"x": 3, "y": 1}, {"x": 3, "y": 3}, {"x": 1, "y": 3}]
    },
    {
      "id": "pick-goal",
      "kind": "goal",
      "label": "Pick goal",
      "points": [{"x": 20, "y": 12}, {"x": 23, "y": 12}, {"x": 23, "y": 15}, {"x": 20, "y": 15}]
    }
  ]
}

terminalShell

iso environment register \
  --project prj_... \
  --name warehouse-reference \
  --version declared-layout-v1 \
  --image-uri ghcr.io/acme/warehouse:v1 \
  --render-manifest environment-render.json

2. Register a versioned perturbation

Use the shared perturbation vocabulary and preserve exact parameter values. Start and end steps describe when the intervention is active; they do not claim when a failure begins.

scenario.pyPython

from iso_obs_schemas import PerturbationFamily, PerturbationSpec

scenario = client.scenarios.register(
    environment_id=environment.id,
    name="delayed-observation",
    version="latency-140ms",
    description="Evaluate delayed perception against the pinned baseline.",
    perturbations=[
        PerturbationSpec(
            family=PerturbationFamily.LATENCY,
            name="observation-delay",
            parameters={"milliseconds": 140},
            start_step=20,
            end_step=160,
        )
    ],
)

TRUTH BOUNDARY

Studio never infers a route, occupancy, physical appearance, trajectory, failure, or causal effect from these declarations. Recorded playback and measured outcomes require run evidence.

Planning

Fail compatibility before you pay for execution.

Compilation catches missing capabilities, unsupported parameters, unsupported artifacts, and simulation-evidence mismatches. Successful compilation produces content-addressed work items and deterministic parallel batches.

compile.pyPython

from iso_obs.simulation import EvidenceUse, ExecutionIntent, compile_simulation_plan

plan = compile_simulation_plan(
    regression_pack,
    adapter,
    intent=ExecutionIntent.REPRODUCTION,
    evidence_use=EvidenceUse.CONFIRMATORY,
)

LEARNED MODELS

Learned-world-model screening requires uncertainty quantification and out-of-distribution detection. Surrogate screening remains discovery-only and cannot certify its own generated failures.

Execution results

Make every worker prove which work item it ran.

A result repeats both the work-item ID and its digest. Missing, duplicate, timed-out, failed, or identity-mismatched results make the campaign incomplete instead of counting as successful system behavior.

worker_result.pyPython

from iso_obs.execution import (
    SimulationRunResult,
    SimulationRunStatus,
    evaluate_simulation_campaign,
    simulation_work_item_digest,
)

result = SimulationRunResult(
    work_item_id=item.work_item_id,
    work_item_digest=simulation_work_item_digest(item),
    run_id=run.id,
    status=SimulationRunStatus.COMPLETED,
    trace=analysis_ready_trace,
    artifact_digests=(replay_digest,),
)

report = evaluate_simulation_campaign(
    plan,
    regression_pack,
    tuple(completed_results),
)

Regression gates

A perfect small sample can still be insufficient.

Gates can require exact conformance or a one-sided Wilson lower confidence bound. Evaluation requires exactly one observation for every declared case and seed, so incomplete execution cannot create a passing rate.

CAMPAIGN CONCLUSION

A passing simulation campaign is evidence within its declared simulator envelope and use. It is not deployment authorization and does not erase sim-to-real or model-form limitations.

Relevant API

Classes and capabilities in simulation execution.

Relevant SDK classes and functions
Class or function	Import from	Use it to
`RegressionPack`	`iso_obs.regression`	Declare canonical, neighborhood, boundary, and control cases plus seeds and gates.
`RegressionCase`	`iso_obs.regression`	Bind scenario parameters, expected outcome, invariants, and role.
`SeedPanel`	`iso_obs.regression`	Declare exact replay, common-random-number, or independent seed strategy.
`GateCriterion`	`iso_obs.regression`	Declare conformance or confidence-bound release criteria.
`SimulationAdapterManifest`	`iso_obs.simulation`	Describe backend identity, evidence, capabilities, and limits.
`CompatibilityReport`	`iso_obs.simulation`	Explain unsupported parameters, artifacts, or required capabilities.
`SimulationExecutionPlan`	`iso_obs.simulation`	Carry content-addressed work items and deterministic execution batches.
`SimulationRunResult`	`iso_obs.execution`	Return identity-bound status, trace, diagnostics, and artifacts for one work item.
`SimulationCampaignReport`	`iso_obs.execution`	Evaluate completeness, identity, domain validity, and regression gates.