Test Execution

Pytest marker policy and local, PR-time, full, and nightly commands.

Note

This page includes implemented local test contracts as well as broader execution guidance. The claim-source-index and scientific-coverage checks below are local, deterministic unit tests; they do not execute a release workflow.

Test execution

Marker

When

Wall-clock

(unmarked)

every PR

~2–5 min

slow

nightly

30–60 min

gpu

manual / GPU runner

varies

spark

nightly (mocked) + weekly (real cluster)

varies

hil

manual (lab-only)

varies

annotator

nightly (mock LLM) + weekly (real LLM)

~10–20 min

golden

nightly

~30 min

 
# PR-time (fast)
pytest -m "not slow and not gpu and not spark and not hil and not annotator and not golden"

# Full local
pytest -m "not gpu and not spark and not hil and not annotator and not golden"

# Nightly
pytest                  # all markers

Claim-source index contract tests

The claim-source index is tested as a validator-bound routing boundary. Run its focused suite with:

PYTHONPATH=src pytest -q tests/unit/test_layer46_claim_source_index.py

The tests construct typed execution-evidence doubles and assert that the builder calls ExecutionEvidenceValidatorV1.validate_bundle before opening an unavailable plan or chapter-guide path. They also require a validator already bound to the supplied resolved paths, manifests bound to the plan and guide bytes, and exact direct dependency coverage for Layers 1-45.

Routing checks are intentionally structural. They reject duplicate or malformed entry-point selector text without loading an entry point, globally duplicated chapter selector rows, and a non-advertised selector that is required, has a status other than UNAVAILABLE, or carries evidence IDs. They do not prove a plugin can import or execute.

The fixture suite covers both source categories. Layer 1-10 rows must remain NOT_ADVERTISED and have no production evidence hashes. Each Layer 11-45 row must match the plan’s chapter and evidence class, status, chapter manifest, chapter gate, and authoritative acceptance gate. Scientific rows require an exact handoff/result/status chain; non-scientific rows reject scientific evidence. Canonical ordering, unique source IDs, and the RFC 8785 index self-hash are also tested.

These tests do not execute module acceptance tests, rerun scientific experiments, assert release readiness, or promote a module’s maturity. They verify only that already-validated evidence is routed without silently inventing, dropping, or relabeling a claim source.

Scientific claim-card coverage tests

Run the focused ClaimCard and ScientificCoverage suite with:

PYTHONPATH=src pytest -q tests/unit/test_layer47_scientific_coverage.py

The suite constructs a complete ClaimCardV1 and verifies frozen, extra-forbidden RFC 8785 self-hashing. It pins Holm at alpha 0.05, 95% confidence intervals, nonempty retained figure and artifact hashes, all six ordered PASS lens outcomes with zero HIGH findings, passing report-author and validation-gate rows, nonreproducing regressions, and explicit deferred architecture metadata. Forged model instances are round-tripped through the wire validator after the Layer 11 context calls, so a modified methodology or evidence field cannot bypass those checks.

Coverage tests require validator context before any card or selector routing: each production source has status-history validation, and a scientific source also has an exact status, manifest, handoff, and result chain validated as a scientific pair. A non-scientific source with science evidence fails rather than receiving a ceremonial claim card.

The selector tests prove exhaustive coverage. Each scientific selector has one card; a scientific module without an advertised selector requires a card keyed by its module name. Missing cards, duplicate cards, duplicate scientific source selectors, FAIL cards, and DEFERRED cards fail coverage. DEFERRED records a known architectural boundary but cannot make an advertised scientific claim pass. The final report must partition expected selectors into passed, failed, and deferred counts, and may be PASS only with no missing, failed, or deferred identifiers.

These unit tests do not rerun a scientific experiment or establish that an individual RF model is physically valid. They verify the coverage and evidence accounting contract after its upstream validation context has already run.

See Also