Counterfactual Behavioral Replay

Counterfactual Behavioral Replay transforms Nomotic's audit trails from static records into an interactive behavioral forensics tool. Take any agent's actual behavioral history and replay it against a different contract version, a different set of invariants, or a different governance configuration, then observe the projected outcomes.

Why Counterfactual Replay

Compliance teams, auditors, and incident responders constantly ask questions that nobody can answer without manual reconstruction:

• "What if we had applied Contract v2.1 instead of v2.0 during last Tuesday's incident?"

• "How many additional denials would the new invariant set have produced over the last 30 days?"

• "If we tighten the semantic anchor threshold from 0.85 to 0.92, which agents would have been flagged and when?"

Counterfactual Behavioral Replay answers these questions automatically. It turns audit conversations from "what happened" into "what would have happened if" — the question every regulator, board member, and risk officer actually wants answered.

How It Works

Behavioral Replay Engine

The BehavioralReplayEngine loads historical action sequences from the audit trail, applies alternative governance configurations, and produces a ReplayReport comparing actual verdicts with counterfactual verdicts.

from nomotic.replay import BehavioralReplayEngine, ReplayConfig
from nomotic.audit_store import AuditStore

engine = BehavioralReplayEngine(audit_store=AuditStore(base_dir))

# "What if we had used stricter thresholds?"
report = engine.replay(
    agent_id="claims-processor",
    replay_config=ReplayConfig(
        allow_threshold=0.9,
        description="Proposed Q2 stricter thresholds",
    ),
)

print(report.generate_summary())
# Replayed 847 actions for claims-processor. The alternative config would
# have produced 23 different verdicts: 18 stricter (more denials), 5 looser.
# Net effect: 2.1% of actions would have been blocked that were previously allowed.

Replay Configuration

The ReplayConfig specifies alternative governance parameters. Any field left as None uses the original configuration:

ReplayConfig(
    contract=alternative_contract,          # BehavioralContract override
    invariants=[{"type": "ratio", ...}],    # InvariantSpec dicts
    dimension_weights={"safety": 2.0},      # Dimension weight overrides
    allow_threshold=0.85,                   # UCS threshold for ALLOW
    deny_threshold=0.25,                    # UCS threshold for DENY
    trust_influence=0.3,                    # Trust influence factor
    drift_thresholds={"action": 0.15},      # Drift threshold overrides
    semantic_anchor_overrides=[...],        # Semantic anchor overrides
    description="Human-readable description",
)

Verdict Comparison

The report highlights divergence points — actions where the alternative configuration would have produced a different verdict:

for dp in report.divergence_points:
    print(f"Action {dp.action_index}: {dp.actual_verdict} -> {dp.counterfactual_verdict}")
    print(f"  Direction: {dp.direction}")  # "stricter", "looser", "lateral"
    print(f"  UCS: {dp.actual_ucs:.3f} -> {dp.counterfactual_ucs:.3f}")

The direction classifies each divergence using a strictness ordering:

Direction	Meaning	Example
stricter	Counterfactual would have been more restrictive	ALLOW → DENY
looser	Counterfactual would have been more permissive	DENY → ALLOW
lateral	Different verdict at same strictness level	ESCALATE → MODIFY
same	No change	ALLOW → ALLOW

Fleet Replay

Replay the same configuration change across multiple agents to understand fleet-wide impact:

results = engine.replay_fleet(
    agent_ids=["agent-1", "agent-2", "agent-3"],
    replay_config=ReplayConfig(allow_threshold=0.85),
)

for agent_id, report in results.items():
    print(f"{agent_id}: {report.total_divergences} divergences")

Configuration Comparison

Compare two alternative configurations against the same behavioral history:

results = engine.compare_configs(
    agent_id="claims-processor",
    config_a=ReplayConfig(allow_threshold=0.8, description="Moderate"),
    config_b=ReplayConfig(allow_threshold=0.9, description="Strict"),
)

print(f"Moderate: {results['config_a'].total_divergences} divergences")
print(f"Strict: {results['config_b'].total_divergences} divergences")

CLI Usage

Replay with threshold overrides

nomotic replay --agent claims-processor --allow-threshold 0.9 --description "Q2 policy test"

Replay with a config file

nomotic replay --agent claims-processor --config replay-config.json --json

Where replay-config.json contains:

{
  "allow_threshold": 0.9,
  "deny_threshold": 0.25,
  "dimension_weights": {"safety": 2.0},
  "description": "Q2 policy proposal"
}

Time-scoped replay

nomotic replay --agent claims-processor --allow-threshold 0.85 \
  --start-time 1709251200 --end-time 1709337600

HTTP API

Replay

POST /v1/replay

Request body:

{
  "agent_id": "claims-processor",
  "replay_config": {
    "allow_threshold": 0.9,
    "description": "Stricter thresholds"
  },
  "start_time": 1709251200,
  "end_time": 1709337600
}

Response: A ReplayReport JSON object with divergence points, counts, and summary.

Compare Configurations

POST /v1/replay/compare

Request body:

{
  "agent_id": "claims-processor",
  "config_a": {"allow_threshold": 0.8, "description": "Moderate"},
  "config_b": {"allow_threshold": 0.9, "description": "Strict"}
}

Response: {"config_a": ReplayReport, "config_b": ReplayReport}

Integration with Drift Taxonomy

Replay integrates with the drift taxonomy: operators can replay a coordinated drift event under alternative governance configurations to determine whether different contract parameters, invariant thresholds, or semantic anchor tolerances would have caught the propagation chain earlier.

# Replay a specific incident time window with tighter drift thresholds
report = engine.replay(
    agent_id="compromised-agent",
    replay_config=ReplayConfig(
        drift_thresholds={"action": 0.10, "target": 0.10},
        allow_threshold=0.85,
        description="Would tighter drift thresholds have caught the propagation?",
    ),
    start_time=incident_start,
    end_time=incident_end,
)

Replay Report

The ReplayReport provides a complete summary:

Field	Description
report_id	Unique identifier for this replay
agent_id	Agent whose history was replayed
actions_replayed	Total actions processed
total_divergences	Actions with different verdicts
stricter_count	Counterfactual was more restrictive
looser_count	Counterfactual was more permissive
divergence_points	List of VerdictComparison objects
summary	Human-readable summary text