PRODUCT QA Engine — Python & Playwright Verification
A local Python verification engine that runs authorized tests, API checks and Playwright browser journeys, then retains evidence for deterministic release decisions.
01 / The problem
Where the friction starts.
A green CI badge or a builder's claim does not establish that a software change is reliable. Projects need repeatable checks, evidence tied to the reviewed revision, and an explicit decision when tests are missing or incomplete.
02 / The experience
A more useful way through.
The local QA Engine V1 enrolls owner-authorized projects, applies policy-defined checks and executes supported Python, npm, HTTP and Chromium journeys. It keeps the observations separate from the release decision rather than allowing generated summaries to certify a build.
A repeatable verification trail helps an engineer review what was actually executed and why a release gate passed or blocked. The available evidence is from local controlled acceptance, not commercial projects or hosted production QA.
03 / The workflow
- 01Enroll an authorized repository and review its allowed commands, targets and required checks
- 02Resolve the change, check policy, and build a deterministic execution plan
- 03Run supported native, API and Chromium browser checks with bounded permissions
- 04Store sanitized evidence and classify failures, retries and incomplete checks
- 05Return PASS, FAIL, REVIEW_REQUIRED or NOT_EVALUATED for human review
Implemented local V1 verification surfaces
Technology & integration areas
Python / Pytest / TypeScript / Playwright / SQLite / FTS5 / CLI / Streamlit
04 / Current evidence
In Development
Fresh execution on 4 November 2026: a controlled three-check fixture passed and was inspected in the real viewer. The full suite recorded 86 passed and 5 failed; browser acceptance remains incomplete.
4 November 2026 — Local Streamlit viewer displaying an existing controlled-fixture run: unit, integration and regression checks succeeded; execution gate PASS, human report review PENDING. The Dry run checkbox configures a new run, not the displayed result.
- Source inspectedRevision used for execution
- Public implementation and tests inspected at 288347d2d38eb4a7bc21aa0f6dd34b911e0331b1 on 4 November 2026. Inspect revision used for execution (new tab)
- Browser inspectedRunning viewer and controlled fixture
- Started the real Streamlit viewer and inspected a newly executed fixture run. Three checks succeeded; the gate returned PASS and the report remained pending human review. Desktop and mobile captures are below.
- PendingBroader execution limitations
- The fresh full-suite run retained five browser-related failures. Hosted execution and independent external-project pilots remain unverified.
05 / My contribution
Decisions I owned.
I developed a policy-driven QA tool around an explicit distinction between implementation and verification. The project uses Python orchestration, an isolated TypeScript browser worker, reviewable result artifacts and a deterministic decision gate, with AI-assisted engineering subject to separate verification.
Technical architecture
Python orchestration resolves the project inventory and policy, dispatches native checks and a separate TypeScript/Playwright browser worker, then stores run records in SQLite and artifacts on disk. A deterministic gate evaluates the recorded execution. The CLI and thin Streamlit viewer use the same Engine API; report approval remains a separate human action.
06 / Engineering decisions
Why the system takes this shape.
- Do not trust builder claims
- Builder statements and prior green CI labels cannot override incomplete or failed mandatory checks. Dry runs are NOT_EVALUATED instead of falsely passing.
- Limit the executor
- Owner enrollment and fixed command IDs bound native execution. API/browser targets are validated; the browser worker uses isolated contexts and bounded access.
- Treat evidence as a release input
- SQLite records checks, attempts and findings; reports and artifacts carry integrity metadata. Flaky and incomplete results require review rather than becoming clean passes.
Implementation / Reviewed source
The logic behind the interface.
Selected exact source excerpts inspected on 4 November 2026. Public repositories link to the executed revision. Private-source selections are authorized for this review draft; final publication approval remains pending.
Gate decisions follow recorded execution
Dry runs do not pass. Evidence integrity and actual check failures are considered before the mandatory-check floor. The full function continues in the linked public source.
qa_engine/gate.py · lines 5–13
def evaluate(run, root):
if run.plan.request.dry_run:
return GateResult(reason_codes=["DRY_RUN"])
if run.run_status in {"RUNNING", "ERROR", "CANCELLED"}:
return GateResult(reason_codes=["RUN_NOT_COMPLETED"])
if any(not Evidence.verify(root, a, run.project_id, run.run_id) for a in run.artifacts):
return GateResult(decision="REVIEW_REQUIRED", reason_codes=["EVIDENCE_INTEGRITY_FAILURE"])
if any(c.status == "FAILED" and c.classification != "FLAKY_SUSPECTED" for c in run.checks):
return GateResult(decision="FAIL", reason_codes=["CONFIRMED_CHECK_FAILURE"])Public source at verified revision · revision 288347d.
Inspect qa_engine/gate.py at 288347d (new tab)Execution is bounded by explicit policy
Required checks, allowed commands and destinations, loopback policy, mutation permission and bounded timeouts are represented explicitly. This is owner-authorized execution, not an untrusted-code sandbox.
qa_engine/domain.py · lines 45–56
class Policy(Model):
version: Identifier = "v1"
required: list[Kind] = Field(min_length=1)
conditional: list[Condition] = []
allowed_commands: list[Identifier] = []
allowed_hosts: list[str] = []
allowed_ports: list[int] = [80, 443]
allow_loopback: bool = False
allow_mutation: bool = False
diagnostic_retries: int = Field(default=1, ge=0, le=2)
timeout_seconds: int = Field(default=30, ge=1, le=300)
max_requests: int = Field(default=100, ge=1, le=1000)Public source at verified revision · revision 288347d.
Inspect qa_engine/domain.py at 288347d (new tab)Verification / 4 November 2026
What actually ran.
Recorded local execution, with fixture scope and unresolved failures kept visible. These results are separate from the portfolio’s own build checks.
python -m pytest -q86 passed · 5 failed · 5 warnings- Fresh full-suite run at public revision 288347d on Windows. Failures: both live axe cases, redirect authorization, broken-login evidence and browser-auth isolation. Mandatory browser/accessibility paths were not successfully verified. This is not a green full-suite result.
Engine.run(RunRequest(project_id=fixture_project))Controlled fixture: PASS · human review PENDING- A newly enrolled owner-controlled repository executed unit, integration and regression checks successfully. The screenshots show that persisted run in the actual viewer. This narrow fixture does not resolve the five full-suite failures.
07 / Limits & learning
What still needs proving.
- Local V1, not a hosted QA service or an untrusted-code sandbox.
- Five full-suite browser-related failures remain unresolved in this environment. The fixture PASS must not be read as full browser or accessibility acceptance.
- No automated repair, merging, production deployment or external messaging was executed.
- The viewer is a thin controller with limited mobile space. A report gate and human export approval are separate states.
What I learned
Trustworthy QA is a policy and evidence problem as much as a test-runner problem. Incomplete required checks, retries and human approvals need explicit states.
08 / Actual screenshots
From the running local application.
Captured directly from a running local application on 4 November 2026 using newly created fictional fixtures. These are actual application screens, not production activity. Open a screen to inspect it; use the arrow keys to move and Escape to close.
4 November 2026 — Local Streamlit viewer displaying an existing controlled-fixture run: unit, integration and regression checks succeeded; execution gate PASS, human report review PENDING. The Dry run checkbox configures a new run, not the displayed result.
4 November 2026 — The same existing fixture run in the real viewer at 390px. This thin Streamlit surface has limited mobile space; no responsive redesign is claimed.