SDK guide / 04

Turn a failure into a map of what to investigate next.

Preserve the original evidence, organize related behavior without forcing a label, locate the reliability boundary, and test whether a smaller condition set still reproduces the failure.

Failure workflow

Do not jump from one trace to one explanation.

01
Package
Bind the observed behavior to reproduction, simulator, analysis, and limitation evidence.
02
Characterize
Compare a reviewed feature signature with known phenotypes while allowing ambiguity and novelty.
03
Map
Estimate reliable, unresolved, and unreliable regions across operating conditions.
04
Select
Spend exploratory compute where severity, uncertainty, and expected information justify it.
05
Intervene
Replay a controlled change while holding the declared nuisance configuration fixed.
06
Minimize
Test which conditions are actually necessary within the executed ablation set.

Portable evidence

Give the failure a durable identity.

A bundle’s content digest identifies the exact serialized evidence. Its failure fingerprint groups stable mechanism factors while excluding summaries, event IDs, seeds, and affected versions.

failure.pyPython

from iso_obs.evidence import EvidenceLevel, FailureEvidenceBundle

failure = FailureEvidenceBundle(
    failure_category="unsafe_stop_margin",
    scenario_family="blind_intersection",
    scenario_version="3",
    perturbation_family="observation_latency",
    invariant_id="minimum-stopping-distance",
    invariant_version="2",
    signal_group=("stopping_margin_m", "velocity_mps"),
    execution_phase="closed_loop_control",
    component="navigation_stack",
    evidence_level=EvidenceLevel.SUSTAINED_DIVERGENCE,
    summary="Stopping margin diverged before the monitor activated.",
    affected_system_versions=("policy-v17",),
    evidence_event_ids=("evt-101", "evt-102"),
    reproduction=reproduction_manifest,
    simulation=simulation_manifest,
    analyses=(divergence_provenance,),
    limitations=("Observed divergence does not establish causality.",),
)

Phenotypes

Classify with a set, not a forced nearest label.

Held-out conformal calibration can return one phenotype, multiple supported phenotypes, or an empty set. The empty set is a novel_candidate and routes the failure to review.

phenotype.pyPython

from iso_obs.failure_phenotypes import (
    FailurePhenotypeQuery,
    classify_failure_phenotype,
)

query = FailurePhenotypeQuery(
    plan_content_digest=plan.content_digest(),
    evidence_digest=new_trace_evidence_digest,
    query_id="incident-0419",
    feature_values=(0.39, 0.11),
)
report = classify_failure_phenotype(plan, model, query)

DISCOVERY ONLY

Phenotype assignments organize investigation. They do not establish mechanism identity and cannot authorize a release decision.

Boundary mapping

Keep the unresolved band visible.

Boundary mapping uses declared dimensions, sampled anchors, and confidence bounds to certify cells only when the evidence supports them. Incomplete design or violated assumptions withholds the map.

boundary.pyPython

from iso_obs.failure_boundaries import (
    fit_failure_boundary_model,
    map_failure_boundary,
)

model = fit_failure_boundary_model(plan, boundary_observations)
boundary_map = map_failure_boundary(plan, model)

RELATED CAPABILITY

Use FailureSurfacePlan and estimate_failure_surface when the primary question is a disaggregated rate across prespecified strata rather than a continuous operating boundary.

Experiment selection

Use compute where it can change what you know.

The selector first satisfies minimum regional coverage. It then combines severity, posterior failure probability, uncertainty, expected variance reduction, population mass, and execution cost.

select.pyPython

from iso_obs.experiment_selection import (
    fit_bayesian_experiment_selector,
    recommend_next_experiment,
)

state = fit_bayesian_experiment_selector(plan, exploratory_observations)
recommendation = recommend_next_experiment(plan, state)
next_region = recommendation.selected_region_id

ADAPTIVE EVIDENCE

Observations chosen adaptively remain discovery-only. After a consequential region is found, define a new fixed design for confirmatory claims.

Mechanism tests

Intervene, then test local minimality.

Counterfactual replay holds the declared scenario, initial state, seed, random stream, simulator evidence, and nuisance configuration fixed. One-minimality is granted only when every single-condition removal was executed and stopped reproduction.

counterfactual.pyPython

from iso_obs.counterfactual import (
    analyze_counterfactual_design,
    assess_minimal_counterexample,
)

report = analyze_counterfactual_design(design, paired_observations)
minimal = assess_minimal_counterexample(
    failure_conditions,
    executed_ablation_trials,
)

Relevant API

Classes and capabilities in failure discovery.

Relevant SDK classes and functions
Class or function	Import from	Use it to
`FailureEvidenceBundle`	`iso_obs.evidence`	Package one failure with reproduction, simulation, analysis, and limitation evidence.
`FailurePhenotypePlan`	`iso_obs.failure_phenotypes`	Declare features, taxonomy, calibration, and target population.
`FailurePhenotypeReport`	`iso_obs.failure_phenotypes`	Return an assigned, ambiguous, or novel-candidate prediction set.
`FailureBoundaryPlan`	`iso_obs.failure_boundaries`	Declare dimensions, anchors, cells, uncertainty, and minimum evidence.
`FailureBoundaryMap`	`iso_obs.failure_boundaries`	Partition cells into reliable, unreliable, or unresolved regions.
`FailureSurfacePlan`	`iso_obs.failure_surface`	Estimate disaggregated failure rates across prespecified strata.
`ExperimentSelectionPlan`	`iso_obs.experiment_selection`	Allocate exploratory compute across declared operating regions.
`CounterfactualDesign`	`iso_obs.counterfactual`	Bind paired replay controls, intervention, and estimand.
`MinimalCounterexampleReport`	`iso_obs.counterfactual`	Assess local one-minimality across executed ablations.