Evidence-Audited Test Engine for an AI Retail App

An AI-guided workflow app had no automated regression capability: every release was validated by hand, and the app's plan-generation quality — is the plan correct, feasible, labor-efficient — was checked by nobody and nothing. Testing on top of a probabilistic CV system also means the instrument itself is noisy: naive assertions read model variance as app bugs.

A test that can only pass is indistinguishable from no test, and an accusation that doesn't survive re-derivation burns trust with the engineering team. So I measured the CV system's run-to-run noise on identical inputs before trusting any reading, and held every finding to an evidence bar of independent re-derivation and an explicit refutation pass before calling it a defect.

Declarative, schema-validated test scenarios distilled from the product ask; a scenario generator that encodes the selection rubric as code; self-verifying fixtures that are pure functions of authored ground truth (validated against a reversed-layout control that correctly fails); a device harness that walks the real app through the full associate workflow; and a deterministic reference-model oracle that checks the generated plan itself for correctness, feasibility, and labor efficiency.

• LLM authors, deterministic code judges: every verdict is reproducible from the evidence package it ships with
• The CV system's run-to-run noise is measured on identical inputs and partitioned out before scoring, so model variance is never recorded as an app bug
• Fixtures that can fail: the 1.0-scoring fixture ships with a reversed-layout control scoring 0.125, so a pass means something
• Findings go to the engineering team as evidence packages a reader can re-derive, framed as questions

• Took an app with no automated regression capability to gated device runs in eight weeks
• Full end-to-end device runs proven, with governed overnight autonomous test sessions: pre-committed stop rules, deny-by-default action vocabularies, zero leaked tasks