A building-automation vendor's web releases went from monthly fear to weekly routine behind a regression harness

Representative examplesEvery case study in this library is an illustrative composite of the kind of engagement we deliver — written to show our method and standards, not to name clients.

Where they started

Building-automation software ages like infrastructure: the web UI this vendor ships with its controllers has grown for a decade across dozens of supported device types, and nobody on staff can say what every screen does anymore. Releases had frozen feature work for a month while QA clicked through two hundred screens by hand, and the last few releases still shipped regressions that reached real buildings — alarms misrouting, schedules not taking. The QA team knows the product cold but has no coding background, and building operators do not forgive a broken schedule the way a phone user forgives a broken button.

What it was costing

The vendor's building-operator web UI had grown for a decade with no automated tests; releases froze features for a month while QA clicked through 200 screens, and two of the last four releases shipped regressions that reached buildings.

What they could see

• Every release froze new feature work for a month while QA manually walked two hundred screens.
• Regressions in alarm routing and schedules reached buildings twice in the last year and generated support escalations.
• The release checklist lived in one QA lead's head; when she was out, releases slipped again.
• Engineers avoided touching the oldest modules because nothing would catch a break until a building reported it.
• Customer-facing release notes described fixes that QA could not verify before the build shipped.

The constraints we worked inside

• The UI talks to simulated and real controllers with different latencies and failure modes — tests must cover both without becoming flaky.
• The QA team knows the product deeply but not code; the harness must be maintainable by them after handover.
• Release cadence had to improve during the engagement, not after it.

What had been tried before

Assigned two developers to write unit tests over the shared business-logic library.

The regressions kept happening at the integration seams — controller latency, session state, permission edge cases — which unit tests on clean inputs never touched.

Bought a record-and-playback UI testing tool and recorded the main operator flows on the simulators.

The recordings broke on every timing difference between the simulated controllers and real hardware, and nobody on the QA team could repair them after playback failures.

What we proposed

We proposed a two-layer harness matched to where risk actually lives: behavioral end-to-end tests over the critical spine — alarms, schedules, setpoints — and visual regression coverage over the long tail of screens where layout regressions hide. The harness had to be QA-native from the first commit: scenarios written in the QA team's vocabulary, page objects they could extend without reading application code, and CI failures that explain themselves in plain terms. The pipeline and preview environments would land in week three, so release cadence improves during the engagement rather than after it.

Just as important is what we ruled out, and why:

• Full behavioral coverage of all two hundred screens — Deep coverage of the long tail would have consumed the whole engagement maintaining brittle flows that rarely break functionally — the risk there is visual, not behavioral.
• A code-centric framework owned by the development team — It repeats the failed pattern: tests the people who understand the product cannot maintain, guaranteeing decay the month the engagement ends.
• Manual QA process improvements without automation — Checklists and parallelism shave days off the cycle but cannot catch a regression class the checklist has never enumerated, which is the failure the vendor just lived through.

How the work ran

Delivered by the systems pod — 2 engineers over 12 weeks, with working increments reviewed with the client every week.

The stack, and the reasoning

Playwright

Its auto-waiting handles the latency gap between simulated and real controllers without the sleeps that made the recorded tool brittle, and tests read like steps, not scripts.

TypeScript

QA staff who did not write the harness extend it every release; the typed page objects are their onboarding — what a fixture needs is declared where the compiler, not a wiki page, confirms it.

GitHub Actions

The vendor's code already lived there, so CI was configuration rather than infrastructure, and per-PR checks arrived without buying or hosting a runner farm.

Docker test rig

Simulated and real controller profiles run as containers with fixed latencies, so the flakiness that killed the previous tool is controlled instead of chased.

Visual regression tooling

Two hundred screens of layout risk cannot be asserted behaviorally; screenshot baselines catch the broken-render class of regressions the manual walkthrough used to miss.

What went wrong

How we worked together

Cadence

A Tuesday harness review with QA and engineering together, plus ad-hoc pairing sessions where QA extended page objects with us watching rather than after we left.

Client side

The QA lead owned scenario vocabulary and release gating; one senior developer owned CI integration; two QA analysts became the harness's day-to-day extenders.

Decisions

What counted as a critical path was decided by QA's release checklist, not by engineering's architecture diagram — the checklist encoded a decade of real failures.

They provided

Access to the controller simulation environment, one real controller rig for the hardware-in-the-loop run, release history, and QA's checklist with its war stories.

What changed

The headline: releases shipping weekly behind the harness, with the two shipped regressions' class caught in ci — Monthly release freeze → weekly cadence, read from CI run history. A second check: qa cycle per release at 12 days → 2 days.

Release day is ordinary work now. The freeze week is gone, features land continuously, and QA spends it investigating the failures the harness found instead of clicking through screens. The QA lead's checklist became a test suite the whole team can read, so the knowledge survives her vacation and eventually her retirement. Engineers touch the old modules again because something other than a building will tell them they broke it. The two shipped-regression classes from the previous year now fail in CI before any operator ever sees them.

The result was read from CI run history against the pre-engagement baseline over the stated window, with a guardrail check on qa cycle per release. Where platform-reported numbers and business outcomes differ, this record says which layer it is quoting.

What they own now

• The harness repository with page objects, fixtures, and QA-authored scenario suites
• The GitHub Actions pipelines for per-PR checks, hardware-in-the-loop runs, and visual baselines
• The Docker controller rig with profiles for simulated and real device families
• A QA-facing authoring guide for adding scenarios and regenerating baselines safely
• Release-gating rules documenting which suites block a build and who may override