Skip to content

End-to-End Regression Testing Strategy

Status: Proposal — for team discussion. 2026-08-25. Scope: the NestJS monolith (this repo, Railway) + the micro-frontends (Cloudflare Pages, dev/prod duplicated). Problem: no QA team; regressions between BE and FE are found by hand or by users. We need automatic, trustworthy regression coverage that a small team can sustain.

0. TL;DR

We already own the hard part: 400 unit specs + 102 API-level e2e specs running against a real Postgres, per-worker DB clones, migration drift gate, deterministic seed with one user per role. That is the domain regression suite. What is missing is narrow and specific:

  1. Nothing catches FE↔BE contract drift before it ships. Our history is full of *-BREAKING.md guides — every one of them is a regression class a CI gate could catch at compile time.
  2. Nothing exercises the composed micro-frontends against the backend. The only "full stack" test is a human clicking around on dev.
  3. Nothing watches production. With no QA, prod is our last line and it is silent.

Proposal, in ROI order — each step is independently useful and shippable:

# Layer Tool Catches Effort
1 API contract gate OpenAPI snapshot + oasdiff in BE CI; generated TS client in MFEs Renamed/removed fields, enum changes, route changes — before merge ~1–2 days
2 Browser E2E, golden paths only Playwright, ~10–25 journeys, dedicated e2e env "The app doesn't work" for a real user in a real browser ~1 week for the first 5 journeys, then incremental
3 Bidirectional triggers GitHub Actions + Railway/Cloudflare deploy hooks BE change breaking FE and FE change breaking on real BE ~1–2 days
4 Production canary Same Playwright journeys, 3–5 of them, every 15 min vs a synthetic prod tenant Outages and prod-only misconfig, minutes after they happen ~1 day

Guiding rule: testing trophy, not pyramid. Many API tests (done), a thin browser layer, a contract gate between the two. Teams without QA who build a fat browser suite drown in flake maintenance within six months. The discipline is keeping layer 2 small.

1. Where we are today

Backend (this repo)

Asset State
Unit specs 400 *.spec.ts (services, controllers, DTOs, policies)
API e2e 102 test/*.e2e-spec.ts via Supertest, full Nest app + real Postgres
E2E isolation test/global-setup-e2e.cjs migrates + seeds a template DB, clones one DB per Jest worker (CREATE DATABASE … TEMPLATE) every run — zero cross-run pollution (ch09 §7)
Seed Two-tier, deterministic. NODE_ENV=local = full destructive seed: four tenants, admin/teacher/referent/student users, RBAC catalogue (ch15)
CI .github/workflows/ci.yml on every PR + pushes to stage: audit → prisma generate → migration drift check → lint → build → unit → e2e. dev/main skip CI and rely on Railway's deploy pipeline (ch10 §9)
Deploy Railway, RAILPACK builder (no Dockerfile), preDeployCommand = migrate deploy && db seed, health check /api/v1/health/ready. Branch → env: dev → development, main → production
API surface Swagger generated from controllers + *.swagger.ts; served at /docs, JSON at /docs-json
Auth HttpOnly Secure; SameSite=None cookies (FE and BE on different domains); Bearer also accepted (ch03)

Micro-frontends

  • Several MFE repos + a shell, deployed to Cloudflare Pages, one project per env (dev / prod).
  • Cloudflare Pages already produces a preview deployment per branch/PR for free — an asset we are not using yet.
  • Test coverage on the FE side: to be filled in by the FE team (component tests? none?).

Gaps

Gap Consequence today
No FE↔BE contract check BE renames a field → FE breaks on dev → someone notices, or doesn't
No composed-system test Shell + remotes + BE + cookies + CORS is only verified by hand
No controllable "today" in the BE Attendance / academic-year journeys can't be made deterministic end-to-end
No deterministic hosted environment dev is shared and mutable; anything automated against it is flaky by construction
No prod monitoring beyond Railway health A broken login on prod is discovered by a school

2. Principles

  1. Test each thing at the cheapest layer that can catch it. Business rules → unit/API specs (already). Shape drift → contract gate. "Does the button work" → browser. Never re-test through the browser what the API suite already proves.
  2. Browser tests are golden paths, not coverage. Target 10–25 journeys total, each answering "can a <role> still do <critical thing>". If a journey isn't something a school would call us about within an hour of it breaking, it doesn't belong here.
  3. Arrange via API, assert via UI. Log in through POST /auth/login, create fixtures through the BE API, then drive only the journey under test through the browser. Clicking through setup screens to reach the screen you actually test is the #1 source of slow, brittle suites.
  4. Deterministic data, every run. Fresh seed, never shared mutable state. We already do this locally and in CI; the hosted e2e environment must do the same.
  5. Flake is a bug with an owner. A flaky test is quarantined the day it flakes and fixed or deleted within a week. Retries hide problems; they don't fix them.
  6. Mirror production topology. Cross-domain cookies, CORS, Cloudflare in front, Railway behind. A suite that runs against localhost proves less than one that runs against real hosts.

3. Layer 1 — API contract gate (start here)

Why first: cheapest, fastest, and it targets our single most frequent regression class. Every entry in docs/fe-guides/*-BREAKING.md would have been a red CI check.

3.1 Backend side

  1. Export the OpenAPI document as a build artifact. A small script (tools/export-openapi.ts) boots the Nest app without listening, runs SwaggerModule.createDocument(app, buildSwaggerConfig()) — the same call main.ts makes — and writes openapi/openapi.json. Commit the file; it is a snapshot of the public contract at that commit.
  2. Diff on every PR. Add a CI step: regenerate from the PR branch, run oasdiff breaking against the committed snapshot on the base branch. Exit non-zero on any breaking change (removed path/field, narrowed enum, required-ness change, type change).
  3. Acknowledged breaks are allowed, silent ones are not. A PR labelled api:breaking passes the gate only if it also touches the authoritative FE guide under docs/fe-guides/ (we already treat a new *-BREAKING.md as a failure mode — the guide is updated in place). The gate makes the existing convention enforceable.
  4. Serve the spec from every environment. /docs-json already exists. The FE tooling below pulls from https://<be-dev>/docs-json.
# .github/workflows/ci.yml — sketch
- name: OpenAPI contract check
  run: |
    npx ts-node tools/export-openapi.ts --out /tmp/openapi.pr.json
    git show origin/${{ github.base_ref }}:openapi/openapi.json > /tmp/openapi.base.json
    npx oasdiff breaking /tmp/openapi.base.json /tmp/openapi.pr.json --fail-on ERR

3.2 Frontend side

  1. Generate the API client from the spec (openapi-typescript for types only, or orval for typed fetchers). Commit the generated file in each MFE.
  2. FE CI regenerates against the latest dev spec and type-checks. A BE rename that landed on dev now breaks the FE build, not the user. The diff of the generated file in the PR is the change review.
  3. Hand-written response interfaces in MFEs are deleted as they get replaced. No parallel truth.

3.3 What about Pact / consumer-driven contracts?

Pact is the heavyweight industry standard for this. It shines when consumers and providers are owned by different teams with different release cadences. For one BE and a handful of MFEs owned by the same team, the OpenAPI schema diff gets ~80% of the value at ~10% of the ceremony. Revisit if a third party ever consumes our API.

4. Layer 2 — Browser E2E with Playwright

4.1 Why Playwright

Industry default since ~2022: free parallelism (no paid cloud needed for sharding), trace viewer for post-mortem on CI failures, auto-waiting (kills most sleep-driven flake), multi-browser, first-class storageState for per-role auth, page.clock for FE time control, and an APIRequestContext for the arrange-via-API pattern. Cypress is fine but pays for parallelism and is Chromium-centric; Selenium is not a serious option for a new suite in 2026.

4.2 Where the suite lives

A dedicated sis-e2e repository. It spans the BE and every MFE, so it belongs to none of them. Both sides trigger it (see §5). Alternative: inside the shell repo — acceptable if the shell is the only composition point, but ownership gets muddy when a BE change breaks it.

4.3 Conventions

  • Auth via API, once per role. A global-setup logs each seeded role in through POST /api/v1/auth/login with an APIRequestContext, saves the resulting cookies as storageState/<role>.json. Every test starts authenticated in ~0 ms. Seeded users must be single-tenant/single-view so login step 1 is terminal; multi-view users (teacher+staff) get the view-selection step scripted explicitly in the helper.
  • Arrange via API. A thin api/ module wrapping the BE calls we need for fixtures (create student, create lesson, …), typed from the same generated client as the MFEs.
  • Locators by role/label/data-testid, never by CSS structure. MFEs add data-testid on interactive elements that journeys need; that is part of the FE definition of done for a feature that gets a journey.
  • No waitForTimeout. Ever. Auto-wait + explicit expect(...).toBeVisible().
  • One journey = one file, named after the user story it protects, tagged by role (@admin, @teacher, @referent, @student) and by tier (@smoke for the PR/canary subset, everything for nightly).
  • Test data prefix. Every entity a test creates carries a e2e-<run-id>- prefix in its name so a failed run's leftovers are identifiable and sweepable, even though the e2e env is reset on every deploy.
  • Time. FE side: page.clock.setFixedTime(...). BE side: see prerequisite in §7 — until a test clock exists, attendance/AY journeys are written against "today" and accept that they are calendar-sensitive.

4.4 Initial journey list (~12, @smoke marked ★)

# Journey Role Protects
1 ★ Login → landing; nav/blocks match /permissions (empty-not-absent hide contract) each seeded role Auth, RBAC → FE visibility
2 ★ Session view switch (teacher ↔ staff) teacher+staff user POST /auth/switch-view + shell re-render
3 ★ Admin creates Department → Grade → Curriculum row admin Config CRUD, cascade UI
4 Admin creates Academic Year (DRAFT) and patches gracePeriodEnding admin AY lifecycle
5 Setup wizard happy path on a tenant left in SETUP by the seed admin Setup state machine end-to-end
6 Student import from .xlsx → row appears in table admin File upload, import pipeline, table lists
7 ★ Teacher marks attendance for today's lesson → counters update teacher Attendance register (calendar-sensitive until test clock)
8 ★ Referent sees child's absence → submits justification → office sees it referent, admin Family loop, notifications inbox
9 Teacher enters a grade → revises it → history visible teacher Grades ledger
10 Admin creates timetable → diagnostics → publish admin Timetables + publish gate
11 Admin sends communication → recipient sees it in inbox admin, teacher Communications + Resend memory/log transport on e2e
12 Disciplinary note → family ack teacher, referent Disciplinary notes
13 Backoffice: platform admin provisions a tenant + first admin platform admin Onboarding flow

Each journey takes a target of < 60 s and the whole @smoke set < 5 min.

4.5 Environment topology

We need a hosted environment that is (a) deterministic, (b) shaped like prod (Cloudflare → Railway, cross-domain cookies), (c) never used by humans.

Recommended: a dedicated e2e environment.

  • Railway: a third environment e2e (BE service + Postgres), auto-deployed from dev like development is. Its seed runs in full-reset mode on every deploy (see §7 prerequisite — today reset mode is tied to NODE_ENV=local; we add an explicit SEED_MODE=full-reset override so the app can still run with the hosted NODE_ENV). CORS_ORIGIN lists the e2e MFE origins. MAIL_TRANSPORT=log or memory — no real email leaves this env.
  • Cloudflare: an e2e variant of the shell (and remotes, if the BE URL is baked at build time) pointed at the e2e BE. This is the same dev/prod duplication mechanism we already run, applied a third time. If the shell can take the BE URL at runtime (window.__ENV__, query param), a single dev build suffices and no MFE duplication is needed — worth checking.
  • Reset before every run, not only on deploy. The E2E workflow triggers a reseed via Railway CLI (railway run --environment e2e -- npx prisma db seed with SEED_MODE=full-reset) so two runs never see each other's data. A GitHub Actions concurrency group serializes runs against the env.

Alternative considered — ephemeral compose inside GitHub Actions (Postgres service container + BE started from source + MFEs built from source). Pros: parallel PR runs, no shared env. Cons: needs every MFE repo checked out or a published build, does not exercise Cloudflare / cross-domain cookies / CORS, slower per run. Keep as a fallback if the hosted env proves unreliable; the suite itself is identical either way — only baseURL changes.

Alternative considered — Railway PR environments. Railway can fork an ephemeral environment per PR (incl. a Postgres copy). Gives true per-PR BE isolation for BE PRs. Costs per-PR compute; worth enabling once the suite is stable and the BE-PR trigger (§5) is the bottleneck.

4.6 Flake policy

  • retries: 1 in CI only, trace: 'on-first-retry', video: 'retain-on-failure'. Locally retries: 0 so flake is felt.
  • Playwright's HTML report + traces uploaded as workflow artifacts on failure; the failing trace link goes in the PR/Slack message, not "tests failed".
  • A test that fails without a code change gets @quarantine the same day; quarantined tests run but don't fail the build, and the tag carries a date. Quarantine older than 7 days is deleted in the weekly review, not "fixed later".
  • You broke it, you fix it. The author of the change that reddened the suite owns the fix, whichever repo the fix lands in.

5. Layer 3 — Triggers (bidirectional)

Our branch model: BE PRs run CI; merges to dev deploy to Railway with no CI; MFE branches get Cloudflare previews. The trigger matrix uses what each platform already gives us.

Event What runs Against How
BE PR Unit + API e2e (today) + contract gate (new) CI Postgres Existing ci.yml
BE merge → dev Playwright @smoke e2e env (auto-deployed from dev) BE repo workflow on push: dev waits until /health/ready on e2e reports the pushed commit SHA (see §7), then repository_dispatch → sis-e2e
MFE PR Playwright @smoke shell e2e + PR preview URL for that remote + BE e2e MFE CI passes its Cloudflare preview URL; shell must support a remote-override (see §7)
MFE merge → dev Playwright full e2e Cloudflare deploy hook / MFE workflow → repository_dispatch
Nightly 03:00 Playwright full, Chromium + Firefox + WebKit e2e schedule in sis-e2e
Prod deploy + every 15 min Canary (§6) prod, synthetic tenant schedule in sis-e2e (or Checkly)

PRs run Chromium only; the nightly covers the other engines. Results post to a Slack channel; a red nightly is triaged first thing in the morning by whoever is on rotation.

6. Layer 4 — Production canary

With no QA and no on-call, production needs to shout for itself.

  • Synthetic tenant on prod, provisioned once through the backoffice like a real school, with a fixed admin/teacher/referent. Never a real school's data.
  • 3–5 read-mostly journeys from the @smoke set: login per role, /permissions-driven nav, open a student, open today's attendance board, open the inbox. Any write must be idempotent and self-cleaning (e.g. create + delete a disciplinary note).
  • Runs every 15 min and immediately after each prod deploy. Two consecutive failures → Slack alert with the trace link.
  • Options: our own sis-e2e workflow on schedule (free, 15-min floor on GitHub cron, jittery), or Checkly which runs Playwright natively with proper alerting and multi-region. Start with the workflow; move to Checkly if alert latency matters.

7. Prerequisites — small changes each side needs to make

Where Change Why
BE tools/export-openapi.ts + committed openapi/openapi.json + oasdiff CI step Layer 1
BE SEED_MODE=full-reset env override (seed today keys reset mode off NODE_ENV=local) Deterministic hosted e2e env without pretending to be local
BE Expose the build commit (Railway injects RAILWAY_GIT_COMMIT_SHA) in /health/ready Lets the BE-merge trigger wait for the right deploy instead of sleeping
BE Test clock: an injectable Clock (now()) defaulting to the system clock, overridable on non-production envs via env var or a header honoured only when NODE_ENV≠production Attendance / AY / grace-period journeys become deterministic. Today there is no such abstraction (new Date() is called inline)
BE A seed tenant left in SETUP state Journey #5
BE CORS_ORIGIN + MAIL_TRANSPORT for the e2e Railway env §4.5
Shell Remote-override mechanism (query param / localStorage key that points one remote at an arbitrary URL, enabled on non-prod builds only) MFE PR previews composed into the shell (§5)
Shell / MFEs Runtime-configurable BE URL, if not already Avoids a third build per MFE for e2e
MFEs data-testid on interactive elements a journey needs; generated API client replaces hand-written types §3.2, §4.3
MFEs Component tests (Vitest + Testing Library) per repo, if absent The middle of the trophy on the FE side; Playwright must not be the only FE test
New repo sis-e2e: Playwright, storageState per role, api/ arrange helpers, workflows for dispatch / schedule / canary Layer 2–4

None of these is large. The test clock is the only one that touches domain code and it should go through the normal design-gate as a cross-cutting concern.

8. Rollout

Phase Deliverable Exit criterion
1 — Contract (week 1) OpenAPI export + snapshot + oasdiff gate in BE CI; one MFE consumes the generated client A deliberate field rename on a branch turns the BE PR red; FE build fails against the renamed spec
2 — Environment (week 1–2) Railway e2e env with reset-on-deploy; Cloudflare e2e shell; SEED_MODE + health SHA in BE A human can open the e2e shell, log in as every seeded role, and the data is identical after a redeploy
3 — First journeys (week 2–3) sis-e2e repo, storageState per role, journeys 1–3 (@smoke), nightly schedule Green nightly for 5 consecutive nights
4 — Triggers (week 3–4) BE-merge dispatch with SHA wait; MFE-PR dispatch with remote override A BE merge to dev produces a Slack result within 10 min; an MFE PR shows a Playwright check
5 — Breadth (ongoing) Journeys 4–13, one per sprint alongside the feature work that touches that area; test clock lands and 7/8 go deterministic @smoke < 5 min, full < 20 min, flake rate < 2% over 30 days
6 — Canary (after phase 3 is stable) Synthetic prod tenant + 15-min schedule + Slack alert A forced prod misconfig (wrong CORS_ORIGIN) is alerted within 30 min

Rule for phase 5 onward: a feature that changes a golden path updates its journey in the same PR (or in the linked sis-e2e PR), exactly as it updates the FE guide today.

9. What we deliberately do not do

  • Mirror the API suite in the browser. RBAC matrices, validation errors, edge cases, temporal rules — all stay in test/*.e2e-spec.ts. The browser only proves the happy path renders and submits.
  • Run automation against development. Shared with humans, mutable, non-deterministic. e2e is separate for a reason.
  • Screenshot/visual regression at the start. Attendance boards and timetables are date-driven; without a test clock every screenshot rots on a calendar boundary. Revisit for a few static pages once the clock exists.
  • Record-and-playback / low-code test tools. Unmaintainable without a QA owner; brittle selectors by construction.
  • MFE-to-MFE contract tests beyond the shared event/prop surface the shell exposes. The composed browser journey covers integration; per-remote unit/component tests cover behaviour.
  • Large retries. retries: 1 in CI is a tripwire for infra blips, not a tolerance for flaky tests.

10. Open questions for the team

  1. MFE stack facts — module federation / single-spa / import maps? Is the BE URL baked at build time or read at runtime? Does the shell already support any remote override? (Decides §4.5 and the shell prerequisite.)
  2. Current FE test coverage — do the MFEs have component tests today? If not, phase 5 should include a minimum bar before we lean on Playwright as the only FE net.
  3. e2e env cost — one more Railway environment (service + Postgres) and one more Cloudflare project per duplicated MFE. Acceptable?
  4. Where does sis-e2e live — new repo (recommended) or inside the shell?
  5. Rotation — who triages the red nightly? A weekly rotation is enough, but it must be named.
  6. Canary tool — GitHub cron (free, jittery) vs Checkly (paid, proper alerting). Fine to start with cron and decide later.
  7. Test clock design — env var vs signed header vs both; goes through the design gate as a cross-cutting concern.

11. References