plg-metrics

Product brief · rendered from docs/PRODUCT.md

plg-metrics: product write-up

Problem

Growth teams make most of their decisions from three artifacts: a funnel, a retention chart, and an A/B test readout. All three are easy to misread, and the errors are systematic rather than random.

The cost is decision error: shipping changes that do nothing, killing changes that work, and building roadmaps on activation definitions that are just proxies for "was already engaged".

Users and jobs to be done

User Job What they need from the tool
Growth PM "Tell me whether to ship onboarding_v2, and how sure I can be." One readout with a decision line, the evidence behind it, and a refusal to answer when the data is broken (SRM).
Growth PM "How long do I need to run this test?" Sample size and duration from a baseline, an MDE, and daily traffic, including the effect of CUPED.
Product analyst "Where is the funnel actually leaking, and for which segment?" Ordered funnel with explicit window semantics, segment breakdowns, and time-to-convert.
Product analyst "What should our activation metric be?" A ranked list of early-behavior rules with coverage, precision, recall, and lift against week-4 retention.
Data scientist "Can I trust these statistics?" Methods tested against scipy, hand calculations, simulation, and synthetic ground truth.
Leadership "Is the product getting stickier?" DAU/MAU and a defined north-star series, weekly.

Scope

In scope (v0.1)

Out of scope (v0.1)

Requirements

  1. Integrity first. The readout runs SRM before any estimate; a mismatch (p < 0.001) overrides the decision to INVALID.
  2. Every estimate has an interval. No point estimate is printed without a CI and a p-value; raw and CUPED estimates appear side by side.
  3. Primary vs secondary is explicit. The primary metric is tested at alpha; secondaries are corrected as a family; any significant regression blocks a ship.
  4. Peeking is visible. The readout states when the always-valid test first allowed a stop and when naive peeking would have stopped.
  5. No silent truncation. Retention cells not fully observed are blank; ratios that need a full 28-day MAU window are blank until it exists.
  6. Reproducible. The same seed produces byte-identical data; Monte Carlo summaries are seeded.
  7. Local and fast. Each CLI command on the 20,000-user, 749k-event synthetic dataset completes in about 1.5 to 2.5 seconds wall time including interpreter start, measured on a 2-vCPU sandbox.
  8. Statistically verified in CI. Stats functions match scipy or hand calculations; interval coverage and mSPRT error control are checked by simulation; the readout must recover the synthetic ground truth.

Success metrics and evals

The tool measures itself in two ways: offline evals that run in this repository, and the product metrics a team would track once it is in use.

Offline evals (shipping gates, run in CI or make validate)

Eval How Current result (bundled synthetic data)
Ground-truth recovery 100 seeds x 20,000 users; does the CUPED 95% CI cover the generator's true effect? 92% activation, 93% active days, 95% paid (one SE is about 2.2 pts)
Estimator bias Mean CUPED estimate minus truth over 100 seeds -0.0008 on a 0.0228 activation effect
Variance reduction Mean CUPED SE / raw SE 0.889 on active days (about 21% less variance)
Peeking control A/A, 30 daily looks, 2,000 sims naive 28.4% false positives, mSPRT 0.9%
SRM detection Generator drops 8% of treatment logs flagged, decision INVALID (test)
A/A sanity Generator with zero lift decision not SHIP (test)
Power calibration Empirical vs predicted power on the primary metric 68% observed vs 70% predicted

Product metrics (once adopted by a growth team)

Minimum viable quality

Thresholds are on the bundled synthetic data and the offline evals above. "Current" is the committed docs/validation/ output and the test suite.

Gate Do not ship Ship Delight Current
SRM detection (8% of treatment logs dropped) not flagged, or decision not INVALID flagged, decision INVALID same flagged, INVALID (test)
A/A sanity any SHIP on a zero-lift dataset never SHIP same not SHIP (test)
Peeking control (A/A, 30 looks, 2,000 sims) mSPRT false-positive rate above 5% at or below 5% at or below 1% 0.9%
CUPED 95% CI coverage, every metric, 100 seeds below 90% (more than 2 SE under nominal) 90% or more 93-97% on every metric 92% / 93% / 95%
Mean CUPED bias, every metric, 100 seeds more than 3 SE of the mean from zero within 3 SE within 1 SE within 1 SE on activation and paid; about 2.3 SE on active days (issue #5)
Power calibration, primary metric empirical more than 10 pts from predicted within 10 pts within 5 pts 68% vs 70%
Variance reduction on the primary, CUPED SE / raw SE above 1.0 (CUPED widens the interval) 1.0 or below 0.90 or below 0.979 (issue #6)
CLI runtime per command, 20,000 users above 10 s under 5 s under 2.5 s about 1.5-2.5 s

v0.1 meets every "Ship" gate. It misses "Delight" on variance reduction for the primary and on active-days bias, which are the two open statistical issues.

Cost at 1x and 10x usage

Estimates, not measurements, built only on this repository's own assumptions: everything runs in-process (DuckDB over Parquet, Python CLI), no paid API, no server, no API keys. Usage here means the size of the event table and how often readouts run.

1x (bundled demo) 10x (estimate)
Data 20,000 users, about 749k events about 200,000 users, about 7.5M events
Per-request API or hosting cost none none
Compute per readout about 1.5-2.5 s per CLI command on 2 vCPUs (measured, see Requirements) not measured; the repo's stated envelope for one machine is tens of millions of events, so 10x stays inside it
make validate (100 regenerated datasets) one batch job on the same machine grows with dataset size; runs offline, not per readout
Where cost starts developer or CI machine time only the next step past this envelope is warehouse connectors (roadmap), where query cost moves to the team's warehouse bill

At 10x the cost that changes is wall time on one machine, not a bill. The repo has no pricing assumptions for a warehouse, so no dollar figure is given for that step.

Trade-offs and alternatives considered

Decision Chosen Alternative Why
Compute DuckDB in-process on Parquet Warehouse (BigQuery, Snowflake) Zero setup, sub-second queries at the bundled scale, and the SQL ports to a warehouse dialect with small changes. The cost is a single-machine ceiling; connectors are the first roadmap item.
Query style .sql files plus one builder ORM or a dataframe API Analysts can read and review SQL; only the funnel needs a builder because its CTE chain depends on step count.
Inference Frequentist primary, Bayesian alongside Bayesian only Frequentist error rates are what stakeholders and correction methods are defined against; P(better) and expected loss answer the question PMs actually ask. Showing both makes disagreement visible instead of hiding it.
Sequential method mSPRT O'Brien-Fleming alpha spending mSPRT needs no fixed number of looks. Alpha spending has more power when the look schedule is known and honored, which in practice it rarely is.
Variance reduction CUPED, one covariate Regression adjustment with many covariates, CUPAC CUPED is transparent and unbiased with a single pre-period metric. Multi-covariate and ML-based adjustment can reduce more variance; they are on the roadmap behind a metric layer that can supply the covariates.
Correction Holm (FWER) default, BH optional Bonferroni, no correction Holm is never less powerful than Bonferroni and gives the same family-wise guarantee; BH suits exploratory readouts with many secondaries.
Activation ranking F1 by default Lift Lift favors rare, extreme thresholds that cover few users; F1 balances precision and reach, which is what an onboarding target needs.
Validation data Synthetic with analytic ground truth Public event datasets No public product dataset ships with a known experiment effect. Synthetic data is the only way to test whether the readout recovers the truth.

Risks

Roadmap

Now (v0.1, this repository)

Next

Later