Observe and compare
Instrument runs, extract aligned signals, compare paired traces, and quantify replicated effects.
client · run · trace · divergence · replicationOpen the guide Reliability Studio / SDK documentation
Start with one run. Follow the evidence as far as it goes.
Instrument your existing evaluation loop, preserve what happened, and build progressively stronger claims without erasing uncertainty or scope.
INSTALL / PYTHON 3.12+
pip install iso-obs- Execution
- Local, CI, simulation, or Studio
- Artifacts
- Versioned, content-addressed JSON
- Default API
- reliability-studio-5cmy6.ondigitalocean.app/api/v1
Quickstart
Wrap the loop you already run.
The run context records the seed, versions, observations, actions, rewards, state, and artifacts. A clean exit marks the run complete; an exception marks it failed and is re-raised.
from iso_obs import ReliabilityClient
client = ReliabilityClient() # reads ISO_OBS_API_KEY
with client.run(
project="robot-arm",
system_version="policy-v17",
environment="warehouse-v4",
scenario="obstructed-pick",
seed=42,
) as run:
obs = env.reset(seed=run.seed)
while True:
action = policy(obs)
next_obs, reward, terminated, truncated, info = env.step(action)
run.step(
observation=obs,
action=action,
reward=reward,
state=info.get("state"),
)
obs = next_obs
if terminated or truncated:
break
run.log_artifact("replay.mp4")CONFIGURATION
Set ISO_OBS_API_KEY. Override the API root with ISO_OBS_BASE_URL. Buffered events flush in batches of 500 and are preserved locally if delivery fails.
Capability map
Choose the question—not a feature bundle.
Begin with the engineering decision you need to support. Each path produces an explicit report with scope, limitations, provenance, and a versioned schema.
Find the boundary
Map supported, unresolved, and unsupported operating regions without forcing a crisp threshold.
failure_boundaries · failure_surface · experiment_selectionOpen the guide Test mechanisms
Run controlled perturbations, paired counterfactuals, and minimal-counterexample analysis.
causal_perturbations · counterfactual · multiplicityOpen the guide Build regression evidence
Package failures, compile simulator-neutral plans, evaluate campaigns, and review promotion.
evidence · regression · simulation · execution · promotionOpen the guide Assess transfer and release
Keep simulator validity, real-world anchors, assurance, and surveillance explicitly scoped.
sim_to_real · transportability · assurance · surveillanceOpen the guide Audit recorded datasets
Ingest JSONL, Parquet, MCAP, and rosbag2 evidence without inferring clocks, labels, or lineage.
dataset_* · replay_capsuleOpen the guide Evaluation examples
Move from observation to controlled evidence.
These are abbreviated from the maintained SDK examples. The linked guides contain the full contracts, required provenance, and interpretation limits.
EXAMPLE 01 / COMPARE
Locate sustained behavioral divergence.
Pair baseline and candidate traces, declare a signal-specific tolerance, and distinguish the first numerical difference from a persistent meaningful change.
Read the full comparison guidefrom iso_obs.divergence import PairingAssessment, SignalSpec, compare_traces
pairing = PairingAssessment.assess(
same_scenario_version=True,
same_environment_version=True,
same_seed=True,
same_perturbation_realization=True,
)
report = compare_traces(
baseline_steps,
candidate_steps,
signals=[
SignalSpec(
path="gripper_force_n",
category="control",
absolute_tolerance=0.5,
persistence_steps=3,
)
],
pairing=pairing,
)EXAMPLE 02 / PLAN
Compile a reproducible simulation campaign.
Preflight adapter capabilities before spending compute, then expand every case and seed into identity-bound work. Confirmatory use remains separate from discovery.
Read the simulation guidefrom iso_obs.simulation import EvidenceUse, ExecutionIntent, compile_simulation_plan
plan = compile_simulation_plan(
regression_pack,
adapter,
intent=ExecutionIntent.REPRODUCTION,
evidence_use=EvidenceUse.CONFIRMATORY,
)Recorded evidence
Bring datasets and robot recordings into the same evidence chain.
Local workflows inspect structure, clocks, labels, synchronization, split contamination, and replay readiness without requiring an API key.
JSONL / PARQUET
# JSONL or Parquet
iso dataset ingest trajectories.jsonl \
--plan ingestion-plan.json \
--output dataset-bundle.json \
--report ingestion-report.json
iso dataset inspect dataset-bundle.json
iso dataset synchronize dataset-bundle.json --plan synchronization-plan.json
iso dataset audit dataset-bundle.json --plan reliability-audit-plan.json
iso dataset split split-request.jsonMCAP / ROS 2 / ROSBAG2
# One MCAP recording
iso dataset ingest recording.mcap \
--plan mcap-ingestion-plan.json \
--output dataset-bundle.json \
--report ingestion-report.json
# Split rosbag2 recording
iso dataset ingest robot-run-42/ \
--plan rosbag2-ingestion-plan.json \
--output dataset-bundle.json \
--report ingestion-report.json \
--recording-report recording-evidence.jsonNO SILENT INFERENCE
Ingestion requires explicit field paths, timestamp units, modalities, clock identifiers, and topic rules. It does not infer labels, clock alignment, split roles, or causal order.
EXAMPLE 03 / REPLAY
Compile a recorded failure into a replay capsule.
Bind the immutable dataset, synchronization report, simulator adapter, and selected failure evidence. Replay fidelity and failure-claim qualification remain independent.
Open the complete replay exampleiso dataset compile-replay replay-request.json \
--bundle dataset-bundle.json \
--synchronization synchronization-report.json \
--adapter simulation-adapter.json \
--output failure-replay-capsule.jsonCLI behavior
Automation can distinguish failure from honest abstention.
Scientific outcomes still emit their complete report. Retain the artifact; do not reduce a run to its process exit code.
| Code | Meaning |
|---|---|
| 0 | Completed; the scientific claim is supported within scope. |
| 1 | API or network failure. |
| 2 | Usage, local input, or authentication failure. |
| 3 | A valid report requires review. |
| 4 | The split audit demonstrates contamination. |
| 5 | The report abstains because evidence is insufficient. |
Evidence guarantees
The SDK preserves what it cannot conclude.
Reliability reports are designed to remain useful when the answer is incomplete, unresolved, or outside the tested envelope.
No point claim alone
Effects travel with intervals, sets, thresholds, or explicit withholding.
No scope-free verdict
Validity envelopes, pairing quality, limitations, and provenance remain attached.
No compensating score
A strong result cannot average away a blocker, open defeater, or missing design requirement.
Full reference