The analysis engine

An analysis engine you can audit line by line.

Every statistic is computed from the raw respondents, not estimated by a model. Every AI-written claim is re-derived from source data before it ships, or it doesn't ship. This page is the methods section: the tests we run, how the AI is used, and the guardrails that keep it honest.

See the guardrails Re-derive a number
In production today Deterministic stats Every number round-trips to source Reproducible at approval

The engine

It computes. It doesn't guess.

Piaza runs the analysis a senior quant would run by hand, deterministically, at render time. No approximations, no model in the loop for the numbers. Here are the methods and the formulas behind them.

Significance

Two-proportion z, pooled

Every crosstab arrow tests a segment against Total. Categorical questions use a χ² test of independence. Bases under threshold are suppressed and flagged, never silently tested.

pooledp̂ = (x₁ + x₂) / (n₁ + n₂)
SE√(p̂(1−p̂)(1/n₁ + 1/n₂))
z(p₁ − p₂) / SE
Confidence intervals

Wilson score, not naive normal

Proportions get the Wilson interval, which stays inside [0,1] and behaves at small n and extreme p. The naive p̂ ± 1.96·SE approximation is never used.

z1.96 (95% CI)
center(p̂ + z²/2n) / (1 + z²/n)
halfz√(p̂(1−p̂)/n + z²/4n²) / (1+z²/n)
NPS

An honest CI, not a proportion test

NPS is promoters minus detractors, so its variance is not a single proportion's. Piaza uses the correct variance for the difference, then a 95% band on the −100 to 100 scale.

Var(p₊ + p₋ − (p₊−p₋)²) / n
margin1.96 · √Var · 100
Weighting

Effective base and design effect

Supplied-column weights are applied consistently, and the effective base is reported so nobody over-reads a weighted n. The AI narrative carries the design effect through, so it never quotes the raw n as if it were the real precision.

n_eff(Σwᵢ)² / Σwᵢ²
deffn / n_eff → e.g. 1.269
Verbatim, quantified

Themes with intervals, not vibes

Open-ends become a proportion: mentioners over answerers, with a Wilson CI. Demographic skew on a theme is a two-proportion z against the base. Theme-to-metric bridges compare mentioners and non-mentioners with a two-sample z on means.

prevmentioners / answerers → Wilson CI
bridge z(x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂)
Banners and question types

Analyst-grade, every question type

Banner plans are segment trees: nested AND / OR / NOT with percentile operators (≥p, ≤p, between-p) and low-base suppression. Scale, likert, CSAT, CES, NPS, numeric, constant-sum, matrix (T2B / B2B / net), rank-order (#1 / top-N / mean rank), and multi-select (respondent % vs response %) each get their correct treatment.

The role of AI

The model narrates. It never sources the numbers.

The division of labour is deliberate and absolute. Deterministic code owns every figure and every quote-to-respondent link. The language model is handed a constrained, enumerated evidence surface and asked only to organise and explain it.

What the LLM does

  • +Drafts theme labels from open-ends, then every theme is re-grounded to specific respondents by code.
  • +Writes the executive summary, findings, and recommendations from an enumerated cite-back table.
  • +Synthesises across questions, surfacing patterns for a human to confirm.
  • +Explains a result in plain English, in the researcher's voice.

What the LLM never does

  • ×Compute a statistic. Every number comes from deterministic code, not a token prediction.
  • ×Attach a quote to a demographic. That is code-level matching: exact substring, then Jaccard ≥ 0.5, else flagged unverified.
  • ×Invent a count, a percentage, or a respondent ID. If it isn't in the cite-back table, it is forbidden in output.
  • ×Supply its own evidence. Evidence is a property of the data, not the prose.
8-agent pipeline Question expertQuality checkCross-Q synthesisData expertData checkerCross-moduleProject reportEvidence auditor

Each agent checks the previous agent's work, and where a deterministic rule is enough, no model is called at all. Scoring that can be exact is exact.

Guardrails we believe in

Six layers between the model and your report.

They work as a stack, not a single check. L1 and L6 stop a bad number upstream, L2 flags soft violations, L4 scores each section, and L5 lets the reader re-run the check. The hard gate is L3: any figure that can't be re-derived from the raw data is rejected, not softened.

L1
Grounded promptThe model only sees enumerated cite-back tables: ground-truth N, per-field N, subgroup N, margin of error. A number not in the tables is disallowed in output.
prompt build
enforced
L2
Heuristic scanRegex flags vague quantifiers ("many respondents" with no proximate number), verbatim over-claims, and any invented methodology N.
post-LLM
flagged
L3
Raw-data round-tripEvery numeric the model emits is recomputed from the respondents. A mismatch beyond tolerance is rejected as unsupported_evidence.
per claim
rejects
L4
Evidence auditorA separate holistic pass scores every section and surfaces issues by severity, feeding the next refinement.
per section
scored
L5
Viewer re-derivationAnyone reading the report can re-run the L3 round-trip against the baked data and watch the number rebuild. That is Show the Math.
in the file
live
L6
Prompt invariant scanA 46-phrase bank blocks forbidden phrasings from ever entering a prompt template, so an over-eager instruction can't be authored in the first place.
authoring
enforced

Ground truth wins over LLM convenience. Always. And insights are frozen at approval, so the report reads identically in ten years.

Nothing is a black box

Click any number. Re-derive it yourself.

Layer 5 is not a marketing promise. It ships in the file. Every significance arrow, interval, and rate opens its own working: inputs, formula in notation, intermediate values, result, and a one-line verdict a researcher can quote.

  • ›Cite the derivation to a stakeholder or a regulator
  • ›Catch a mismatch the moment the number looks wrong
  • ›Hold AI-written figures to the same standard as computed ones
SIGNIFICANCE ▲ · REVEAL
inputsp₁=0.62 (n₁=180)   p₂=0.48 (n₂=420)
pooledp̂ = 0.524
SE√(0.524·0.476·(1/180+1/420)) = 0.043
z(0.62 − 0.48) / 0.043 = 3.26
p0.001 → significant at 99%
Verdict: segment over-indexes vs Total. Real, not noise. Base is healthy (n₁=180).

In production today

Everything on this page ships in every file.

Not a roadmap. These run at generation time on real projects, and travel inside the delivered report.

✓Crosstab significance with low-base suppression
✓Wilson CIs on every proportion
✓Banner plans: nested AND/OR/NOT, percentile ops
✓Verbatim prevalence + demographic-skew z
✓Theme-to-metric bridges (two-sample z)
✓NPS with a correct variance band
✓Weighting with effective base and deff
✓Norms and targets, blank ≠ zero
✓Matrix T2B/B2B/net, rank-order, constant-sum
✓Raw-data round-trip on every AI numeric
✓Deterministic quote-to-respondent matching
✓Show the Math reveal on every marker

Kick the tyres on a real file.

Open a generated report, click a number, and watch it rebuild from the raw respondents.

Or email hello@piaza.ai