AI Insight Engine: Deterministic Where It Matters
A production LLM pipeline for retail execution data where the model never computes a dollar figure
- Shown here
- architecture and evaluation results — no customer data, figures as magnitude classes
- Numbers from
- the enforcement stack's own run tracker and adversarial backtest artifacts
- Scope
- the insight pipeline and its policy layer; the platform around it was engineering's
Category managers pay consultancies for revenue-opportunity narratives — where the loss is, why, what to do — assembled by hand across many dashboards. An LLM can write that narrative in minutes, but a fabricated number in a revenue document is disqualifying. The engineering problem wasn't generation; it was making generation trustworthy.
Testing surfaced the failure class that guardrails themselves can cause: a schema constraint forced the model to fabricate a store count of 2,155 on an insight where no store-level data existed. The judge structurally couldn't catch it — the schema demanded a number, so the model invented one. That reframed the design: constraints have to match the data's actual grain, and every number needs a provenance the model can't override.
A two-pass pipeline: deterministic signal scanners find candidates; the LLM's first pass selects drill-downs from a finite registry (it cannot invent queries); a formula engine computes every dollar with auditable, sensitivity-ranged formulas; materiality and coherence gates filter; the LLM's second pass writes narrative under a versioned rule set; an LLM judge scores against a weighted rubric and regenerates on failure. The 2,155 fabrication was fixed with a per-insight-type rule override — verified 2,155 → 1.
The output schema and the data disagreed about what existed. The model resolved the disagreement by inventing something — and the judge could not catch it, because the schema had demanded exactly that field.
- store_count — integer, minimum 1
- required on every insight type
- validator checks presence and type
- …and nothing checks whether the number came from anywhere
- manufacturer-total year-over-year brief
- aggregate grain — no store rows at all
- no field the count could be derived from
- model emitted: 2,155 stores
Fixed with a per-insight-type override that pins the count for aggregate-grain insights, plus a brief directive. Re-verified on the same paid harness: 2,155 → 1, no fabricated store identifiers. A constraint that does not match the data's grain is not a guardrail, it is an instruction to invent.
- The LLM never computes a dollar figure — drill selection and narrative composition only; a deterministic formula engine owns every number
- Quality rules live as versioned data in YAML and markdown rather than in code, so a rule change is measurable by re-running the eval suites with no deploy
- Adversarial eval before trust: a 12-case suite of known-bad outputs put the enforcement stack at 8 of 12 rejected, below its own 80% gate; root-causing two false accepts and tightening the rules brought it to 10 of 12
- Schema constraints must match data grain — the fabrication the schema itself forced (2,155 phantom stores) became a per-insight-type override and a standing eval case
- The first full real-data run cleared the substrate checks with no enforcement violations and no regenerations
- Adversarial rejection measured, root-caused, and improved on a 12-case suite: 8 of 12 → 10 of 12 against its own 80% gate
- Identified six-figure per-brand revenue opportunities and an eight-figure annualized decline flag, sized as opportunity for the category team to act on
- Complete engineering handoff package (code, semantic layer, methodology, README) delivered to the platform team