Skip to content

Testing architecture

Owner: Quill (QA Engineer) Status: living document — implementation companion to ADR-0010.

See ADR-0010: Test architecture for the architectural decision. This page documents the implementation: fixture API, env vars, debugging tips, examples. Commitments (Real-Postgres rule, fixture scope choices, category boundaries, CI contract, what is deliberately NOT in V2) live in the ADR; the how-to-use lives here. Both stay in sync; if they diverge, the ADR wins and this page is updated.

This page is the single source of truth for how RBO V2 is tested: which kinds of tests live where, what each fixture does, and the two non-negotiable rules. New tests added to the repo are expected to fit into the patterns described here; if a new test does not fit, file an issue rather than invent a parallel pattern.

The two non-negotiable rules

Rule 1 — tests must hit a real Postgres database

Mocks of the database, the SQLAlchemy session, or any layer between application code and Postgres are forbidden. Use a live Postgres instance for every test that touches the data layer.

Origin. A prior incident saw mocked tests pass while a production migration failed. The cost of "tests are fast because they mock the DB" is "tests are decorative and bugs ship to prod." We are not repeating that. The fixture module enforces this rule at the entry point: pytest will not run without an explicit live-DB URL.

What this means in practice:

  • Unit tests for pure functions (no DB access) are fine and stay unmocked because they have nothing to mock.
  • Tests for code that reaches the DB (handlers, repositories, ORM models, the tenancy chokepoint, receipt generation that touches rows) must run against a real Postgres connection through the db_session fixture.
  • "But it's just a tiny SELECT, surely a mock is fine" is exactly the reasoning that produced the original incident. No carve-outs.

Rule 2 — TEST_DATABASE_URL is mandatory; no silent fallback

The database_url fixture (tests/conftest.py) raises RuntimeError if TEST_DATABASE_URL is unset, and again if it points at port 6432 (the conventional PgBouncer port).

Why no localhost fallback. Falling back to postgresql://localhost:5432/postgres would let the suite "pass" against whatever DB happens to exist on the developer's laptop -- including stale schemas from other projects. A loud failure forces an explicit choice.

Why no fallback to the runtime DATABASE_URL. That env var points at PgBouncer per the keystone platform contract (see the PgBouncer constraint memory). Reusing it for tests would silently break the per-test transactional rollback that db_session relies on -- see the next section.

Why direct Postgres, not PgBouncer, in tests

The keystone platform routes RBO's runtime DATABASE_URL through PgBouncer at 127.0.0.1:6432. PgBouncer is a hard part of the production contract. For tests, we deliberately bypass it.

The db_session fixture uses the SQLAlchemy "external transaction" test pattern: open one connection, BEGIN an outer transaction, bind the session to that connection with join_transaction_mode="create_savepoint", ROLLBACK on teardown. This works iff every query in the test runs on the same physical connection.

PgBouncer in transaction-pool mode (the platform's mode) does not guarantee that. Two queries inside one application-level transaction can land on different backend connections. The SAVEPOINT pattern then either errors or, worse, silently leaks state across tests.

Tests therefore point at the direct Postgres port (usually 5432). The pooler is exercised in production; it is not exercised in unit / integration tests. End-to-end tests against a live deployment do exercise the pooler, by definition (they go through Caddy and the real env).

The test pyramid for RBO

Tier Owner Scope Tooling Where it runs
Unit Pax (BE) / Juno (FE) for their own code Pure functions; no I/O; no DB. pytest for backend; vitest (or chosen FE runner) for frontend. Local + CI test:unit.
Integration Pax / Juno for their own code One slice through the app: handler -> service -> repo -> real Postgres. The db_session and client fixtures are the standard tools. pytest + httpx.AsyncClient + real DB. Local + CI test:unit.
Tenancy regression Quill (QA) -- writes the suite; Pax adds new tenant-scoped models to the enumeration Every tenant-scoped table refused unscoped reads; every list/show endpoint exercised with two stringers and asserted leak-free. pytest + db_session. Local + CI test:unit (gate).
Receipt regression (golden file) Quill PDF/HTML diff for the receipt template; layout regressions blocked. TBD (likely PDF text extraction + normalised diff). Local + CI test:unit (slot reserved; not implemented until M14).
End-to-end Quill Browser + real backend + real DB. Highest-priority M-items. Playwright (likely choice; see team/qa_engineer.md). CI e2e stage (future).

Quill does not own unit + basic integration tests. Pax and Juno write tests for their own code. Quill owns the regression / end-to-end / tenancy layers. See team/qa_engineer.md for the boundary.

Fixture API

All fixtures live in tests/conftest.py. Import-by-name; pytest auto-injects.

Fixture Scope Purpose
database_url session Returns TEST_DATABASE_URL or raises. Use directly only in fixture composition; tests should depend on db_engine / db_session instead.
db_engine session Async SQLAlchemy AsyncEngine bound to the test DB. Reused across the whole pytest session. Disposes on teardown.
db_session function Per-test AsyncSession with the SAVEPOINT-rollback pattern. Test code can call session.commit() and the data is still rolled back at teardown.
client function httpx.AsyncClient driving the FastAPI app via ASGITransport. No live socket; in-process.

Example — handler test

import pytest
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession

pytestmark = pytest.mark.asyncio

async def test_orders_list_for_stringer_a(
    db_session: AsyncSession,
    client: AsyncClient,
) -> None:
    # arrange: seed via db_session (real INSERTs against real DB)
    # act: client.get("/orders", headers={"Authorization": ...})
    # assert: response shape, no rows from stringer B
    ...

Example — tenancy regression (pattern only; lands when models exist)

The pattern (per ADR-0003 / ADR-0004): enumerate every tenant-scoped model class via reflection, open a session without binding current_stringer_id, attempt SELECT * FROM <table> LIMIT 1, assert that the chokepoint refuses the query.

async def test_every_tenant_scoped_table_refuses_unbound_reads(
    db_session: AsyncSession,
) -> None:
    from app.db.tenancy import iter_tenant_scoped_models  # Pax's lane

    for model in iter_tenant_scoped_models():
        with pytest.raises(TenancyNotBoundError):
            await db_session.execute(select(model).limit(1))

This test gates merges to main once Pax has the chokepoint implemented. It is intentionally reflective: adding a new tenant-scoped table without registering it is a test failure, which is the point.

Golden-file pattern slot (V2 M14, V3 onwards)

When M14 (receipt delivery) lands, the receipt regression suite follows this shape:

  • Render a known receipt against fixed inputs.
  • Extract text from the rendered PDF (or normalise rendered HTML).
  • Diff against a checked-in golden file under tests/golden/receipts/.
  • A diff fails the test; updating the golden requires explicit --update-goldens flag + reviewer sign-off in the MR.

This slot is reserved here so neither Pax nor Juno reinvents shape-detection ad-hoc. Implementation lands when receipts ship, not before.

CI shape

The test:unit job in .gitlab-ci.yml runs on every MR + main push:

  • image: python:3.12-slim-bookworm
  • services: [postgres:16-alpine] (alias postgres, trust auth)
  • TEST_DATABASE_URL: postgresql://...:5432/... (direct port)
  • script: pytest -ra

Inline definition rather than a keystone CI brick because the v5 .test_migrate_smoke brick bundles a migration step and RBO has no migrations yet. If/when this shape stabilises, file a keystone issue to extract a brick.

What the mkdocs nav says

This page lives under Architecture. ADR-0010 is the formal record; this page is its implementation companion -- the fast-moving fixture API stays here; the durable principles are crystallised in the ADR.

Cross-references