Take one evaluation from code to inspectable evidence.

The path

Five steps, one evidence chain.

1. 01
   Install

   Add the Python SDK and CLI to the environment that already runs your evaluation.

2. 02
   Identify

   Create a project and register the system version whose behavior you are measuring.

3. 03
   Instrument

   Wrap the existing environment loop in a RunContext and record each interaction.

4. 04
   Preserve

   Attach metrics and artifacts that make the behavior interpretable and reproducible.

5. 05
   Inspect

   Review the completed run locally, through the CLI, or in Reliability Studio.

Step 01

Install and establish identity.

The SDK reads ISO_OBS_API_KEY from the environment. The CLI can store the same key in a mode-600 configuration file. Creating a project writes iso-obs.toml in the working directory.

pip install iso-obs
pip install iso-obs-cli

# Generate an actionable local report immediately
python -m iso_obs.quickstart

# Store an API key for the CLI
iso auth login --api-key <YOUR_KEY>
iso auth whoami

# Create a project and local iso-obs.toml
iso project init --name "robot-arm"

Steps 02–04

Record the loop without moving the system.

Your policy, simulator, and environment remain where they are. The context records ordered observations, actions, state, rewards, and metrics. Every event emitted for one interaction shares a deterministic step index.

from iso_obs import ReliabilityClient

client = ReliabilityClient()  # ISO_OBS_API_KEY

with client.run(
    project="robot-arm",
    system_version="policy-v17",
    environment="warehouse-v4",
    scenario="obstructed-pick",
    seed=42,
) as run:
    observation = env.reset(seed=run.seed)
    while True:
        action = policy(observation)
        next_observation, reward, terminated, truncated, info = env.step(action)
        run.step(
            observation=observation,
            action=action,
            reward=reward,
            state=info.get("state"),
            metrics={"tracking_error": info["tracking_error"]},
        )
        observation = next_observation
        if terminated or truncated:
            break
    run.log_artifact("replay.mp4")

Step 05

Inspect the run and retain its provenance.

Registering a system version binds human-readable versioning to an artifact URI and optional source commit. Run inspection then exposes the captured context and metrics without asking the evaluation code to recreate them.

# Register a version when you need explicit model provenance
iso system register \
  --project prj_1234567890abcdef12345678 \
  --name "robot-arm-policy" \
  --version "17.0.0" \
  --artifact-uri s3://models/policy-v17.pt

# Inspect the completed run and its metrics
iso run inspect run_1234567890abcdef12345678

Operational behavior

Telemetry remains recoverable when delivery fails.

Requests retry up to three times on rate limits, server failures, and transport errors. A failed flush keeps its buffer for the next attempt. If the final close still cannot deliver, pending events are appended to .iso-obs/pending-events.jsonl.

Relevant API

The classes you use first.