standard_asr.compliance
The compliance test suite. Engine authors run these checks to verify their plugin
before publishing; applications can also run check_entrypoints() at startup to
catch broken installations early.
from standard_asr.compliance import check_entrypoints, check_streaming_param_gatingCompliance helpers for Standard ASR plugin authors.
ComplianceIssue#classSource
class ComplianceIssue
ComplianceIssue(
level: Literal['error', 'warning'],
code: str,
message: str,
model: str | None = None,
)Single compliance issue detected during validation.
Mirrors the runtime Diagnostic shape: every
issue carries a stable, machine-readable code so a CI pipeline can
assert against (or whitelist) a specific category without string-matching the
human-readable message -- the message is for humans and MAY be
reworded, the code is the programmatic contract (the same reasoning that
gives Diagnostic a code).
| Name | Type | Description |
|---|---|---|
level | Literal['error', 'warning'] | Issue severity ( |
code | str | Stable machine-readable category identifier (for example,
|
message | str | Human-readable description (for display; MAY be reworded). |
model= None | str | None | The model key the issue is attributed to, or |
ComplianceReport#classSource
class ComplianceReport
ComplianceReport(
registry: ModelRegistry | None,
issues: list[ComplianceIssue],
)Aggregate result returned by the compliance check functions.
| Name | Type | Description |
|---|---|---|
registry | ModelRegistry | None | Model registry the entry-point check ran against. The
behavioral checks ( |
issues | list[ComplianceIssue] | Collected compliance issues. |
passed | bool | Return |
iter_level#methodSource
def ComplianceReport.iter_level(
level: Literal['error', 'warning'],
) -> Iterable[ComplianceIssue]Yield issues matching level.
| Name | Type | Description |
|---|---|---|
level | Literal['error', 'warning'] | Severity level to filter. |
Iterable[ComplianceIssue]Iterable of matching issues.
DEFAULT_SYNC_BRIDGE_TIMEOUT#attributeSource
DEFAULT_SYNC_BRIDGE_TIMEOUT = 5.0SupportsCapabilities#classSource
class SupportsCapabilities(Protocol)The one-method surface check_sync_bridge needs from an engine.
A deliberately minimal protocol instead of StandardASR: the check
consults nothing but supports(), and demanding the full surface would
force every caller with a partial test double (or a wrapper) to fake
members the check never touches. (StandardASR itself is now
strict-assignable from real plugins -- config is a read-only protocol
property -- so this narrowing is about least-surface, not a typing
workaround.)
SupportsWireRecommendation#classSource
class SupportsWireRecommendation(Protocol)The two-member surface check_recommended_wire_format needs.
A deliberately minimal protocol instead of EngineBase: the check's
subjects include structural (non-EngineBase) engines -- the standard's
own promise -- and the previous EngineBase-typed signature invited
calling EngineBase-only members on them (the check once called
ensure_stream_format_supported, not a StandardASR member, so a
fully compliant structural engine failed with a false
recommended_wire_format_self_inconsistent verdict on an
AttributeError).
| Name | Type | Description |
|---|---|---|
properties | BaseProperties |
recommended_wire_format#methodSource
def SupportsWireRecommendation.recommended_wire_format() -> AudioFormat | NoneReturn the engine's recommended minimal wire format.
AudioFormat | NoneThe recommended format, or
Nonewhen none is derivable.
assert_prefix_invariant#functionSource
def assert_prefix_invariant(events: Iterable[TranscriptionEvent]) -> NoneAssert a recorded stream's partials honor the frozen-prefix invariant.
Test helper for engine authors. Partials are lossy under backpressure: the
base coalesces pending partials when the consumer is slow, so the
partial count is non-deterministic -- the same engine may surface five
partials or none purely by consumer timing. Asserting a count is therefore
flaky; assert the invariant instead. This checks only the prefix invariant
-- a segment's frozen prefix (text[:stable_until]) is never rewritten and
stable_until never regresses -- across however many partials survived
coalescing, and (unlike check_event_sequence) does NOT require a
terminal event, so it also applies to a mid-stream slice. It replays events
through the same runtime _LifecycleGuard the
runtime uses, so the assertion cannot drift from enforcement.
| Name | Type | Description |
|---|---|---|
events | Iterable[TranscriptionEvent] | The recorded events, in emission order. |
AssertionErrorIf any segment's frozen prefix was rewritten or its
stable_untilregressed.
check_entrypoints#functionSource
def check_entrypoints(
registry: ModelRegistry | None = None,
*,
strict_discovery: bool = False,
instantiate: bool = True,
names: Iterable[str] | None = None,
) -> ComplianceReportValidate that discovered entry points conform to expectations.
Environment-level invariants are reported as issues alongside per-engine ones -- never as raised exceptions -- so a single command yields one report even when discovery itself found problems:
- Engine-identity collisions (an
engine_idprovided by more than one distribution, makingconfig.enginerouting ambiguous) are reported as errors here. The discovery layer only marks them (registry.shadowed_engine_ids); the compliance suite is the fail-loud layer the standard mandates for them, so a collision is a compliance failure even on a default (non-strict) run rather than a log line. - Invalid entry-point names surface as an error too. When
strict_discoveryisTrue,discover_modelswould normally raise on them; this function catches that and converts it into an error issue (so itsRaises: Nonecontract holds and a report is always returned), then re-discovers leniently so the valid engines are still checked.
| Name | Type | Description |
|---|---|---|
registry= None | ModelRegistry | None | Optional pre-discovered registry. When provided, discovery is
skipped and |
strict_discovery= False | bool | Treat invalid entry-point names as a hard discovery
error (reported as an error issue here, never raised). Default
|
instantiate= True | bool | If |
names= None | Iterable[str] | None | Restrict the PER-ENGINE checks to these model keys; |
ComplianceReportCompliance report summarizing findings.
check_event_sequence#functionSource
def check_event_sequence(
events: Iterable[TranscriptionEvent],
*,
allow_empty: bool = False,
capabilities: DeclaredCapabilities | None = None,
) -> ComplianceReportValidate a recorded streaming event sequence against the invariants.
Behavioral check for streaming engines that is pure: it replays an already-captured event stream through the standard lifecycle/frontier guard and reports every invariant it violates, without ever instantiating or calling an engine. (Behavioral checks that would require running a model -- strict sample-rate, the input-conversion matrix, language membership -- are deliberately left to unit tests, because invoking a cloud engine from a compliance run would be a billable side effect; this one only inspects data the author already produced.)
Detected violations (each an error): an illegal lifecycle transition
(partial/final after a segment is finalized/superseded; a non-closed
final after final; superseding a closed segment), a non-monotonic
stable_until or audio_processed_until, a rewritten frozen prefix, the
full supersede invariants -- frozen-prefix preservation
across a replacement (the concatenated frozen text of old_ids MUST be
preserved by new_ids), old_ids that were never announced, a
new_id that reintroduces an already-known segment, and an empty
new_ids (pure deletion) that would destroy frozen text -- an event stream
that never reaches a terminal (done / non-recoverable error) event,
an empty sequence (unless allow_empty=True), and any event emitted
after the session-terminal event (a terminal MUST be the last event).
The per-segment lifecycle / frozen-prefix / supersede checks are obtained by
replaying the events through the same
_LifecycleGuard the runtime uses, so the
compliance verdict cannot drift from the runtime's enforcement. Events after
the session-terminal are flagged and not replayed (they do not exist in a
well-formed stream, so they MUST NOT mutate segment state).
| Name | Type | Description |
|---|---|---|
events | Iterable[TranscriptionEvent] | The recorded events to validate, in emission order. |
allow_empty= False | bool | When |
capabilities= None | DeclaredCapabilities | None | When provided, additionally cross-check each event against
the engine's declared streaming capabilities: a stream MUST NOT
exceed what it declares -- for example, emit a non-zero |
ComplianceReportclass:
ComplianceReport;passedisTruewhen the sequenceComplianceReporthonors every streaming invariant.
check_provider_params_swap_safety#functionSource
def check_provider_params_swap_safety(engine: StandardASR) -> ComplianceReportAssert an engine always rejects another engine's provider_params.
The standard makes provider_params swap-safety an unconditional MUST:
a wrong-typed provider_params (the classic "switched engines, forgot to
change the params model" bug) MUST raise
InvalidProviderParamError independent of
strict / best_effort -- it is a code bug, not a capability negotiation, so
it is never silently dropped. The EngineBase template enforces this
in gate_params before any audio is decoded or the model is touched, so
an engine that bypassed the template and forgot the check is the gap this
probe closes -- the same "bypassed the template must show up here" reasoning
behind check_streaming_param_gating.
The probe calls the engine's public
transcribe
with a foreign ProviderParams subclass private to this module (so it
can never be the engine's own declared type) and a one-sample silent input.
Because provider-params validation precedes audio decoding and inference, the
probe incurs no billable side effect under either policy. The contract is the
same for a strict and a best_effort engine: it MUST raise
InvalidProviderParamError.
| Name | Type | Description |
|---|---|---|
engine | StandardASR | The engine instance to exercise (any policy). |
ComplianceReportclass:
ComplianceReport;passedisTruewhen the engine raisedComplianceReportInvalidProviderParamErrorfor the foreign provider params.
check_recommended_wire_format#functionSource
def check_recommended_wire_format(
engine: SupportsWireRecommendation,
*,
model: str | None = None,
) -> ComplianceReportAssert an engine's recommended wire format is one it would itself accept.
recommended_wire_format is the
single source of truth for the minimal wire AudioFormat the standard
layer opens a streaming_input session with when the application chose none
-- the CLI sync-bridge runner and the streaming gating probe both rely on it.
A self-inconsistent engine, whose recommended format the standard
session-establishment rule rejects for its own declared Properties, would
make those paths fail-loud on a format the standard layer chose rather
than the application -- a silent-looking compliance trap. This closes that
loop: when a format is recommended it MUST pass
ensure_wire_format_supported -- the
pure (Properties, AudioFormat) rule that
ensure_stream_format_supported
itself implements. Validating via the pure rule (never the EngineBase
method) keeps the verdict correct for structural engines, which have no
such method.
| Name | Type | Description |
|---|---|---|
engine | SupportsWireRecommendation | The engine under test. Deliberately NOT required to declare
|
model= None | str | None | The model key ( |
ComplianceReportclass:
ComplianceReport.passedisTruewhen no format isComplianceReportrecommended, or the recommended format is accepted by the engine.
check_streaming_param_gating#functionSource
def check_streaming_param_gating(engine: StandardASR) -> ComplianceReportAssert a streaming engine gates an unsupported standard parameter.
Closes the streaming-gating bypass gap as a compliance failure rather
than a silent one: the base
start_transcription
template runs gate_params(mode="streaming") for every engine, so a
"forgot to gate" engine (one that bypassed the template) must show up here.
The check establishes a streaming session (via start_transcription,
which constructs the session but does not enter its context, so no wire
connection is opened) for the first standard parameter the engine does
not support in streaming mode and asserts the standard contract:
- strict policy -- the call MUST raise
UnsupportedFeatureErrorwhoseparamidentifies the gated field; - best_effort policy -- the call MUST succeed, drop (or degrade) the
parameter, and surface the probe's expected diagnostic (for example,
unsupported_parameter_ignored) viasession.diagnostics().
When the engine supports every probed parameter at the feature level, the
check falls back to violating a declared sub-constraint of a supported
feature (a prompt over its max_tokens budget, or a word-timestamp
granularity outside the declared granularities; see
_pick_sub_constraint_probe) and asserts the same strict-raise /
best_effort-diagnose contract.
Legal session context. A streaming_input engine is probed with a
valid wire AudioFormat taken from the engine's own
recommended_wire_format
(guarded like every other sync-member call), so an engine that legitimately
fail-louds on a missing audio_format is not misjudged as
non-compliant for obeying the standard. A streaming_output-only engine is
probed with a one-sample silent audio input, but only under the strict
policy: strict gating raises before the audio is decoded or the model is
touched (gate order: params first, then audio), so the probe is free of the
billable side effect a best_effort probe would incur by reaching the engine.
A best_effort streaming_output-only engine is therefore reported as a
warning skip (inconclusive) rather than driven into real inference.
Distinguishing a gating raise from "streaming unsupported". The strict
contract is satisfied only by an UnsupportedFeatureError whose param
equals the probed field. An engine that declares a streaming axis but never
implements the hook raises an UnsupportedFeatureError with no (or a
different) param; that is a capability lie, not a gating success, and is
recorded as a distinct error instead of being mistaken for a clean pass.
An engine that declared streaming support yet accepts the violating
parameter -- the "forgot to gate" engine that bypassed the base template --
is a compliance failure here, so the gap is loud rather than silent. An
engine that raises anything other than UnsupportedFeatureError from the
probe is likewise recorded as a compliance error (never re-raised), so one
crashing engine cannot abort the run.
Engines that declare no streaming support, or that support every probed parameter and declare no violable sub-constraint, yield a clean (no-op) pass -- there is nothing to gate.
| Name | Type | Description |
|---|---|---|
engine | StandardASR | The engine instance to exercise. Its |
ComplianceReportclass:
ComplianceReport;passedisTruewhen the engine gatedComplianceReportthe unsupported parameter per its policy (or had nothing to gate).
check_sync_bridge#functionSource
def check_sync_bridge(
session_factory: Callable[[], TranscriptionSession],
*,
timeout: float = DEFAULT_SYNC_BRIDGE_TIMEOUT,
model: str | None = None,
engine: SupportsCapabilities | None = None,
) -> ComplianceReportDrive an async engine's SyncSession from an external thread.
Implements the standard's sync-bridge mandate: a no-deadlock / no-leak
test. A fresh session is created and driven synchronously from a different
thread than the one that built it, feeding no audio and immediately ending
input. The test asserts the session terminates (emits a terminal event and
tears down) within timeout -- a deadlock or a leaked background loop/
thread shows up as a timeout.
| Name | Type | Description |
|---|---|---|
session_factory | Callable[[], TranscriptionSession] | A zero-argument callable returning a fresh async
|
timeout= DEFAULT_SYNC_BRIDGE_TIMEOUT | float | Seconds granted to EACH phase of the check independently:
session establishment ( |
model= None | str | None | The model key ( |
engine= None | SupportsCapabilities | None | The engine the factory drives, if available (anything with a
|
ComplianceReportclass:
ComplianceReport.passedisTruewhen the bridgeComplianceReportterminated cleanly with no leaked background loop thread, or when the
ComplianceReportcheck is not applicable (session establishment refused as unsupported
ComplianceReportby an engine KNOWN not to declare
streaming_input; reported as aComplianceReportsync_bridge_not_applicablewarning, never as an engine failure).ComplianceReportAn
UnsupportedFeatureErrorfrom anywhere PAST establishment (theComplianceReportengine's
_open,end_audio, event drain, close) is always aComplianceReportfailing
sync_bridge_raised-- the not-applicable carve-out isComplianceReportscoped to the factory call alone.
ValueErrorIf
timeoutis not finite or not strictly positive (a caller code bug, rejected independent of any policy).
check_transcription_result#functionSource
def check_transcription_result(
result: TranscriptionResult,
*,
capabilities: DeclaredCapabilities,
) -> ComplianceReportCross-check a recorded batch result against the declared capabilities.
Behavioral check for batch engines that is pure, mirroring
check_event_sequence: it inspects a result the author already
produced -- it never instantiates or calls an engine (invoking a cloud
engine from a compliance run would be a billable side effect). In v1 it
verifies the diarization couple: a result carrying speaker labels anywhere
(top-level segments[] / words[], a segment's words, or any
channels[i] view) while batch.diarization is unsupported (or the
batch domain is absent -- fail-closed) is a capability⇄result desync,
reported as an error with code result_exceeds_diarization.
Honest scoping note: this is the only diarization coverage the suite
can offer. Positive diarization behavior -- labels present when
requested, correctly attributed, consistent across views -- is unverifiable
without multi-speaker audio fixtures, and the standard probes feed silence.
The check is therefore an opportunistic negative cross-check an author
runs over their own recorded results; an always_on engine necessarily
declares supported=True (the model validator forbids the contradictory
pair) and is never flagged here.
| Name | Type | Description |
|---|---|---|
result | TranscriptionResult | The recorded result to validate. |
capabilities | DeclaredCapabilities | The engine's declared capabilities (pass
|
ComplianceReportclass:
ComplianceReport;passedisTruewhen the result doesComplianceReportnot exceed the declared capabilities.
prepare_requires_arguments#functionSource
def prepare_requires_arguments(prepare: Callable[..., object]) -> boolReturn whether a prepare() warm-up hook needs caller-supplied arguments.
A warm-up hook MUST be invocable with no arguments. A parameter
makes the hook non-conforming only when it is required: it has no default
and is positional-or-keyword, positional-only, or keyword-only. *args and
**kwargs impose no required argument, and a bound method's self is
already supplied, so neither counts.
This is the single definition of the zero-argument half of the contract,
shared by _check_prepare_hook and the standard-asr prepare
CLI command so the compliance verdict and the runtime behavior cannot drift.
| Name | Type | Description |
|---|---|---|
prepare | Callable[..., object] | An engine's |
boolTruewhen callingprepare()with no arguments would fail because aboolrequired parameter is unfilled;
Falsefor a valid zero-argument hookbool(or one whose signature cannot be introspected).
validate_bridge_timeout#functionSource
def validate_bridge_timeout(timeout: float) -> floatValidate a sync-bridge timeout: MUST be finite and strictly positive.
The single owner of the rule, shared by check_sync_bridge and the
CLI's --bridge-timeout parser (which wraps the ValueError into an
argparse usage error) so the two layers can never drift: <= 0 yields
an instant false "did not terminate" verdict against a compliant engine,
inf/nan hangs the check on the very deadlock it diagnoses, and a
finite value above threading.TIMEOUT_MAX would blow up as an
OverflowError out of Thread.join / Future.result mid-check --
a validated timeout MUST be one the bridge's waits can actually take. No
clamping: silently shortening a caller's timeout would be an implicit
rewrite of an explicit value.
| Name | Type | Description |
|---|---|---|
timeout | float | The candidate timeout in seconds. |
floattimeoutunchanged.
ValueErrorIf
timeoutis not finite, not strictly positive, or exceeds this platform'sthreading.TIMEOUT_MAX.