Skip to content

API Reference

Recording and asserting

agentsnap.core.recorder.AgentRecorder

Context manager that records an agent run and writes a golden snapshot.

Intercepts every LLM/tool call made through an active adapter or PatchSet while the context is open, then writes {snapshot_dir}/{test_name}.json on clean exit (nothing is written if the block raises). Supports both sync (with) and async (async with) usage.

Usage::

with PatchSet():
    with AgentRecorder("my_agent", model="claude-haiku-4-5") as rec:
        rec.input_data = {"query": "hello"}   # optional metadata
        result = my_agent("hello")
        rec.output = result
# Writes __agent_snapshots__/my_agent.json

agentsnap.core.asserter.AgentAsserter

Context manager that replays an agent run and compares against the snapshot.

On first use (no snapshot file), automatically records the run as the golden instead of raising SnapshotNotFoundError. Subsequent runs assert against it.

Snapshot read is deferred to exit so that self.input (set inside the with block) can drive auto-hash scenario resolution before the file is looked up.

agentsnap.patches.PatchSet

Context manager that monkey-patches all installed LLM SDKs.

Intercepts LLM calls at the SDK class level so any client — wrapped or unwrapped — is captured by an active TraceAccumulator. Patchers for SDKs that are not installed are silently skipped.

.. warning:: Do not combine PatchSet with agentsnap adapters (e.g. AnthropicAdapter) on the same client. Both the adapter and the PatchSet interceptor will fire, silently recording every LLM call twice. Use one or the other, not both.

Usage::

with PatchSet():
    client = anthropic.Anthropic()   # no AnthropicAdapter needed
    with AgentRecorder("my_test") as rec:
        rec.output = my_agent(client, "query")

Diff engine

agentsnap.core.diff.LLMJudge

Semantic scorer that uses an LLM to compare outputs instead of embeddings.

More accurate than cosine similarity for factual content — understands that "Paris is in France" and "France contains Paris" are equivalent, and that "Python 3.9" vs "Python 3.12" is a meaningful factual change.

Works with any OpenAI-compatible endpoint (OpenRouter, OpenAI, etc.).

Usage

from agentsnap.core.diff import LLMJudge judge = LLMJudge(api_key="sk-or-...", base_url="https://openrouter.ai/api/v1") with AgentAsserter("my_test", judge=judge) as a: ...

from_env() classmethod

Return a configured LLMJudge if AGENTSNAP_JUDGE_API_KEY is set, else None.

Reads model and base_url from env vars or [tool.agentsnap] in pyproject.toml.

score(old, new, key=None)

Return a 0.0-1.0 equivalence score for two text outputs.

if given, reasons are stored under this key (e.g. "llm_call[0]", "output")

so AgentRegressionError can look them up by step name. If None, falls back to a sequential counter key.

score_structural(old_tools, new_tools)

Score whether a tool sequence change is a meaningful behavioral regression.

Returns (score, reason). Score 1.0 = equivalent, 0.0 = fundamentally different. Does not increment _call_count or populate _reasons (structural is a separate concern).

agentsnap.core.diff.DiffConfig dataclass

Consolidates all comparison settings passed to compute_diff.

agentsnap.core.diff.DiffReport dataclass

Result of compute_diff() — one entry per comparison layer.

failed_checks contains strings like "structural", "model_tools", "model_tool_args", "arguments", "llm_requests" (replay mode), or "semantic:output" / "semantic:llm_call[n]" — one per layer/step that failed. passed is True iff failed_checks is empty.

Exceptions

agentsnap.exceptions.AgentRegressionError

Bases: Exception

Raised by AgentAsserter when the new trace drifts beyond threshold.

str(exc) is a human-readable multi-section report (structural, argument, model-tools, and semantic diffs). Programmatic callers should inspect exc.diff_report, a DiffReport with structural_diff, argument_diffs, semantic_scores, semantic_reasons, and failed_checks.

agentsnap.exceptions.SnapshotFormatError

Bases: Exception

Snapshot file cannot be used for the requested operation.

Raised when replay mode is requested on a snapshot recorded before raw responses were captured (version 1.0 files).

agentsnap.exceptions.ReplayError

Bases: Exception

Replay diverged from the recording.

Raised when the agent makes more LLM calls than the snapshot contains, tool call order changes under replay_tools=True, or the provider does not support replay yet.

agentsnap.exceptions.SnapshotNotFoundError

Bases: Exception

Raised when no snapshot file exists for a test (direct SDK use only).

AgentAsserter catches this internally and auto-records a golden run instead of propagating it; it only reaches calling code when a snapshot is looked up directly (e.g. read_snapshot()).