Testing strategy

​

The pyramid (4 layers + evals)

​

1. Unit — packages/schema, packages/core pure logic

​

2. Integration — the workhorse layer

• Runs against real Postgres + pgvector — never a mock, never SQLite. Locally: the same docker-compose.yml postgres as self-host (test database). CI: separate ephemeral Neon branches per selected integration shard (ci.yml integration (...) jobs aggregated by test), with the heavy DB package split into db-structure, db-search, db-auth, and db-memory slices.
• Two isolation strategies, used deliberately (packages/db/test/integration/helpers.ts):
  • Savepoint pattern — withTestTransaction() for tests that own their SQL: rolled back, fast, parallel-safe.
  • Truncate-based cleanup — resetDomainTables() for suites exercising the real withTenant() path: withTenant commits on its own pool, so cross-connection visibility makes savepoints inapplicable there. Honest trade: those suites serialize on cleanup. Because these global TRUNCATEs are not isolated across packages on a shared database (--fileParallelism=false only serializes within a package), the root test:integration script pins turbo --concurrency=1 for local/single-branch runs. CI keeps package-internal serialization, but gives each selected shard its own Neon branch. DB sub-shards also get separate branches, so their truncate-heavy suites cannot collide with each other.
• RLS tests run as the runtime role, not the owner role (owner bypasses RLS — a green isolation test as superuser proves nothing). The test harness provisions roles via the same scripts/provision-roles.sql as production.
• Mandatory suites that exist from the start (packages/db/test/integration/):
  • Tenant isolation: user A writes, user B queries — across every query path including edges, facts, commitments, proposals. Mirrors the eval slice but as fast deterministic tests.
  • withTenant discipline: concurrent pooled connections cannot leak app.user_id.
  • Append-only enforcement: UPDATE/DELETE on memory_events/audit_log fails at the grant level; no DELETE anywhere in the memory domain.
  • Migration round-trip: fresh DB → all migrations → structure parity (tables, policies+NULLIF, FSM trigger, HNSW index, SECURITY DEFINER resolvers).
  • HNSW parity: pgvector vs the eval gate’s exact cosine on the real golden-set embeddings — the bridge between the eval gate’s upper bound and what production serves. Two-regime (see Data model & RLS): natural plan asserted index-driven (selective tenants get btree+exact — correct), plus a forced-HNSW arm (enable_sort=off, hnsw.iterative_scan) with EXPLAIN-asserted index use; both ≥ 0.95 overlap.

​

3. Contract — MCP and REST surfaces

• Spin up the express app in-process (SDK 1.x + express); drive MCP tools through the official MCP client SDK (not hand-rolled JSON-RPC) so transport, session, and structured-output semantics are actually exercised.
• Every tool: happy path + auth-missing + cross-tenant + output-schema validation (the response must parse against the tool’s declared outputSchema — enforced as plain tests, not a bespoke CI job).
• REST mirrors: generated OpenAPI is diffed in CI; a contract change shows up in the PR.

​

4. E2E — minimal by design

​

LLM policy in tests

• Layers 1–4 never call a real model. packages/llm ships a FakeGateway (deterministic embeddings via seeded hash → vector; canned completions per operation key). Tests assert on our logic, not model behavior.
• Model behavior is the eval harness’s job — that’s the separation: tests prove the machine is wired right; evals prove the memory is good. A PR can’t compensate for a golden-set regression with passing unit tests, and vice versa.
• Recorded-fixture mode (capture a real gateway response once, replay) is allowed for extraction-shaped logic, stored under __fixtures__/, refreshed deliberately — never auto-recorded in CI.

​

Test health policy

1. No skipped tests on main. it.skip/todo fails the lint step. A test that can’t pass is either fixed in the PR or deleted with a linked issue — a pile of dormant markers is how a suite rots.
2. No flake tolerance: a test that fails intermittently is quarantined by deletion + issue, not retry-until-green. CI has no retry-on-failure config.
3. Coverage is observed, not gated. Vitest coverage reported on PRs for review judgment; no threshold ratchet (thresholds breed assertion-free tests). The mandatory-suite list above is the real floor.
4. Speed budgets are enforced: unit <30s, integration <3min, contract <2min, total ci.yml <10min wall. A PR that blows the budget fixes it or splits the suite — slow CI is how the job sprawl started.
5. Tests-with-the-change: every behavior PR carries its tests; there is no “test backfill” epic, ever.

​

Local developer loop

pnpm test              # unit only — no services needed
docker compose up -d postgres redis
pnpm test:integration  # integration + contract against local PG
pnpm --filter @3ngram/eval run gate   # the exact per-PR blocking slice (no network)