# A/B Testing & Experimentation Statistics API > Frequentist, Bayesian, sequential and CUPED A/B test analysis, sample ratio mismatch, sample size and MDE, multiple-testing correction, delta-method ratio metrics, bandit allocation, deterministic bucketing, feature-flag rule evaluation, novelty detection, segment lift, guardrail and holdout checks. Pure computation on the numbers you send. Each paid operation uses a fixed USDC price on Base with the x402 exact scheme. Unpaid GET, HEAD, and POST return 402 payment terms (POST validates the body only after payment terms are satisfied). A request that carries payment-signature is not yet charged: settlement happens only after a successful (2xx) result. HTTP 4xx/5xx are not charged. Coverage bodies with status complete, partial, or failed all return HTTP 200 and are charged at the full listed price. Settled 2xx responses include a payment-response header; 4xx/5xx do not. Follow https://www.brainiac.software/agent-instructions.txt for install, payment pins, idempotency, and billing rules. - Product: https://www.brainiac.software/api/experimentation - Docs: https://www.brainiac.software/api/experimentation/docs - OpenAPI: https://www.brainiac.software/api/experimentation/openapi.json - Contact: support@brainiac.software (https://www.brainiac.software/contact) ## Operations ### POST /api/v1/experimentation/ab-test-frequentist $0.02 USDC. Two-arm A/B scorecard for conversion (two-proportion z) or continuous (Welch t) metrics: absolute and relative lift, confidence intervals, p-value and significance. Pure computation on supplied aggregates. Inputs: metricType, alpha?, alternative?, control, treatment Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-ab-test-frequentist Example request: ```json { "metricType": "proportion", "alpha": 0.05, "alternative": "two-sided", "control": { "users": 10000, "conversions": 1000 }, "treatment": { "users": 10000, "conversions": 1100 } } ``` Example response: ```json { "controlRate": 0.1, "treatmentRate": 0.11, "absoluteLift": 0.01, "absoluteCi": [ 0.001504059132, 0.01849594087 ], "relativeLift": 0.1, "relativeCi": [ 0.01087011253, 0.1891298875 ], "statistic": 2.306634774, "pValue": 0.02107518927, "significant": true, "test": "two-proportion z (pooled SE for test, unpooled SE for CI)", "sources": [ { "source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)", "url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "Deng, Knoblich, Lu (2018) Applying the Delta Method in Metric Analytics", "url": "https://arxiv.org/abs/1803.06336", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Normal approximation: unreliable when expected successes or failures per arm are below about 10. Relative-lift interval…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/bayesian-ab $0.03 USDC. Posterior, probability to beat control, probability of being best, expected loss and equal-tailed credible intervals for 2-10 conversion arms. Deterministic: exact Miller sum or quadrature, no Monte Carlo. Inputs: prior?, arms, credibleLevel? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-bayesian-ab Example request: ```json { "prior": { "alpha": 1, "beta": 1 }, "arms": [ { "name": "A", "users": 10000, "conversions": 1000 }, { "name": "B", "users": 10000, "conversions": 1100 } ], "credibleLevel": 0.95 } ``` Example response: ```json { "arms": [ { "name": "A", "posterior": { "alpha": 1001, "beta": 9001 }, "mean": 0.100079984, "credible": [ 0.09427511307, 0.106036332 ], "probBeatControl": null, "probBest": 0.01054264559, "expectedLoss": 0.01001358271 }, { "name": "B", "posterior": { "alpha": 1101, "beta": 8901 }, "mean": 0.1100779844, "credible": [ 0.1040187816, 0.1162848782 ], "probBeatControl": 0.9894573544, "probBest": 0.9894573544, "expectedLoss": 0.00001558231261 } ], "control": "A", "credibleLevel": 0.95, "method": { "probBeatControl": "exact (Miller 2015)", "probBest": "pairwise probability to beat control", "expectedLoss": "deterministic Simpson quadrature of E[max_j p_j] minus posterior mean" }, "sources": [ { "source": "Miller (2015) Formulas for Bayesian A/B Testing", "url": "https://www.evanmiller.org/bayesian-ab-testing.html", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Independent Beta-Binomial posteriors; no Monte Carlo, so results are reproducible. Quadrature spans the 1e-9 to 1-1e-9 …", "Probabilities describe the posterior under the supplied prior, not the long-run error rate of a procedure.", "…" ] } ``` ### POST /api/v1/experimentation/sequential-test $0.04 USDC. Lan-DeMets group-sequential boundaries (O'Brien-Fleming or Pocock spending) for planned looks with a stop/continue decision, or an always-valid mSPRT p-value for continuous monitoring. Inputs: mode, alpha?, spending?, informationFractions?, observedZ?, estimate?, variance?, tau2? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-sequential-test Example request: ```json { "mode": "group_sequential", "alpha": 0.05, "spending": "obrien_fleming", "informationFractions": [ 0.2, 0.4, 0.6, 0.8, 1 ], "observedZ": [ 1.2, 2.1, null, null, null ] } ``` Example response: ```json { "boundaries": [ 4.876884949, 3.357011922, "…" ], "cumulativeAlpha": [ 0.000001077742526, 0.0007883035134, "…" ], "decision": { "look": 2, "crossed": false, "boundary": 3.357011922, "observedZ": 2.1 }, "spending": "obrien_fleming", "sources": [ { "source": "Lan & DeMets (1983) Discrete sequential boundaries for clinical trials, Biometrika 70(3)", "url": "https://doi.org/10.1093/biomet/70.3.659", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Two-sided symmetric boundaries; the total type I error is alpha and each side spends alpha/2. Boundaries assume the pla…", "Numerical recursion on a Simpson grid (at least 2,001 points) with bisection to 1e-9; agrees with published Lan-DeMets …", "…" ] } ``` ### POST /api/v1/experimentation/cuped-adjust $0.03 USDC. CUPED adjustment with a pre-experiment covariate on unit-level data: theta, variance reduction, and Welch tests on raw and adjusted values. Inputs: units, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-cuped-adjust Example request: ```json { "units": [ { "group": "control", "y": 11, "x": 10 }, { "group": "treatment", "y": 13, "x": 12 }, { "group": "control", "y": 9, "x": 9 }, { "group": "treatment", "y": 17, "x": 15 }, { "group": "control", "y": 12, "x": 11 }, { "group": "treatment", "y": 15, "x": 14 }, { "group": "control", "y": 9, "x": 8 }, { "group": "treatment", "y": 14, "x": 13 }, { "group": "control", "y": 10, "x": 10 }, { "group": "treatment", "y": 14, "x": 12 } ], "alpha": 0.05 } ``` Example response: ```json { "theta": 1.18018018, "varianceReduction": 0.9602708298, "raw": { "diff": 4.4, "se": 0.894427191, "pValue": 0.001242936514, "ci": [ 2.32934137, 6.47065863 ], "significant": true }, "adjusted": { "controlMean": 12.32432432, "treatmentMean": 12.47567568, "diff": 0.1513513514, "se": 0.3536446295, "pValue": 0.679972045, "ci": [ -0.6643475398, 0.9670502425 ], "significant": false }, "groups": { "control": "control", "treatment": "treatment" }, "unitsWithoutCovariate": 0, "sources": [ { "source": "Deng, Xu, Kohavi, Walker (2013) CUPED: improving sensitivity of controlled experiments", "url": "https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)", "url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "theta is estimated on both groups pooled, as in the CUPED paper; the covariate must be measured before assignment or it…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/sample-ratio-mismatch $0.01 USDC. Chi-square goodness-of-fit test of observed arm counts against expected traffic shares, with an exact binomial p-value for tiny two-arm counts and generic causes when a mismatch is flagged. Inputs: arms, threshold? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-sample-ratio-mismatch Example request: ```json { "arms": [ { "name": "control", "observed": 50000, "expectedShare": 0.5 }, { "name": "treatment", "observed": 49000, "expectedShare": 0.5 } ], "threshold": 0.0005 } ``` Example response: ```json { "chiSquare": 10.1010101, "df": 1, "pValue": 0.001481880775, "srm": false, "threshold": 0.0005, "arms": [ { "name": "control", "observed": 50000, "expected": 49500, "deviationPct": 1.01 }, { "name": "treatment", "observed": 49000, "expected": 49500, "deviationPct": -1.01 } ], "hints": [], "sources": [ { "source": "NIST/SEMATECH e-Handbook: chi-square goodness-of-fit test", "url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "A non-significant result does not prove assignment is balanced; the default threshold 0.0005 is conservative on purpose.", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/sample-size-mde $0.01 USDC. Required users per arm and duration for a target minimum detectable effect, or the MDE for a fixed sample, for proportion or mean metrics with power, multi-arm Bonferroni and unequal allocation. Inputs: metricType, baseline?, baselineSd?, baselineMean?, mde?, mdeType?, nPerArm?, alpha?, alternative?, power?, arms?, allocation?, dailyUsers?, correction? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-sample-size-mde Example request: ```json { "metricType": "proportion", "baseline": 0.1, "mde": 0.1, "mdeType": "relative", "alpha": 0.05, "power": 0.8, "arms": 2, "allocation": [ 0.5, 0.5 ], "dailyUsers": 2000, "correction": "bonferroni" } ``` Example response: ```json { "nPerArm": 14749, "nByArm": [ 14749, 14749 ], "totalN": 29498, "days": 15, "alphaPerComparison": 0.05, "formula": "unpooled normal approximation", "sources": [ { "source": "NIST/SEMATECH e-Handbook: sample sizes required for means", "url": "https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Normal approximation; for small samples or rare events check the result by simulation.", "Allocation index 0 is the control; every other arm is compared with it.", "…" ] } ``` ### POST /api/v1/experimentation/multiple-testing-correct $0.01 USDC. Bonferroni, Holm and Benjamini-Hochberg adjusted p-values and reject decisions across metrics or arms, returned in input order. Inputs: pValues, methods?, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-multiple-testing-correct Example request: ```json { "pValues": [ { "id": "m1", "p": 0.01 }, { "id": "m2", "p": 0.04 }, { "id": "m3", "p": 0.03 }, { "id": "m4", "p": 0.005 } ], "methods": [ "bonferroni", "holm", "benjamini_hochberg" ], "alpha": 0.05 } ``` Example response: ```json { "results": [ { "id": "m1", "p": 0.01, "bonferroni": 0.04, "holm": 0.03, "benjamini_hochberg": 0.02, "rejected": { "bonferroni": true, "holm": true, "benjamini_hochberg": true } }, { "id": "m2", "p": 0.04, "bonferroni": 0.16, "holm": 0.06, "benjamini_hochberg": 0.04, "rejected": { "bonferroni": false, "holm": false, "benjamini_hochberg": true } }, "…" ], "alpha": 0.05, "sources": [ { "source": "Holm (1979) A simple sequentially rejective multiple test procedure", "url": "https://www.jstor.org/stable/4615733", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "Benjamini & Hochberg (1995) Controlling the false discovery rate", "url": "https://doi.org/10.1111/j.2517-6161.1995.tb02031.x", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Bonferroni and Holm control the family-wise error rate; Benjamini-Hochberg controls the false discovery rate and assume…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/delta-method-ratio $0.03 USDC. Confidence interval for a ratio metric whose analysis unit differs from the randomisation unit, plus the treatment-vs-control difference and relative lift. Accepts unit rows or aggregates. Inputs: arms, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-delta-method-ratio Example request: ```json { "arms": [ { "name": "control", "units": [ { "num": 10, "den": 2 }, { "num": 0, "den": 1 }, { "num": 5, "den": 1 }, { "num": 20, "den": 4 }, { "num": 3, "den": 1 }, { "num": 0, "den": 2 }, { "num": 8, "den": 2 }, { "num": 12, "den": 3 } ] } ], "alpha": 0.05 } ``` Example response: ```json { "arms": [ { "name": "control", "ratio": 3.625, "se": 0.6931585317, "ci": [ 2.266434242, 4.983565758 ] } ], "difference": { "estimate": null, "se": null, "ci": null, "pValue": null }, "relativeLift": null, "relativeCi": null, "sources": [ { "source": "Deng, Knoblich, Lu (2018) Applying the Delta Method in Metric Analytics", "url": "https://arxiv.org/abs/1803.06336", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "First-order delta method with a normal quantile; it is accurate for large numbers of randomisation units and unreliable…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/bandit-allocate $0.03 USDC. Next-period traffic weights from arm outcomes: Thompson-sampling probabilities by deterministic quadrature, or UCB1 indices, with an exploration floor. Inputs: policy, arms, prior?, minShare? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-bandit-allocate Example request: ```json { "policy": "thompson", "arms": [ { "name": "A", "successes": 100, "failures": 900 }, { "name": "B", "successes": 120, "failures": 880 }, { "name": "C", "successes": 90, "failures": 910 } ], "prior": { "alpha": 1, "beta": 1 }, "minShare": 0.05 } ``` Example response: ```json { "policy": "thompson", "weights": [ { "name": "A", "probBest": 0.07474923213, "weight": 0.07176986341 }, { "name": "B", "probBest": 0.9146879375, "weight": 0.8782301366 }, "…" ], "sources": [ { "source": "Russo et al. (2018) A Tutorial on Thompson Sampling", "url": "https://arxiv.org/abs/1707.02038", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "Miller (2015) Formulas for Bayesian A/B Testing", "url": "https://www.evanmiller.org/bayesian-ab-testing.html", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Thompson weights are exact posterior probabilities of being best (quadrature), not random draws, so repeated calls retu…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/bucket-assign $0.01 USDC. Stateless SHA-256 bucketing of unit ids into an experiment with traffic gating, ordered group shares and an optional layer, documented so any SDK can reproduce it. Inputs: unitIds, experiment, layer? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-bucket-assign Example request: ```json { "unitIds": [ "user_42", "user_43" ], "experiment": { "salt": "exp_checkout_v2", "trafficShare": 1, "groups": [ { "name": "control", "share": 0.5 }, { "name": "treatment", "share": 0.5 } ] }, "layer": null } ``` Example response: ```json { "assignments": [ { "unitId": "user_42", "bucket": 7765, "groupBucket": 8754, "inExperiment": true, "group": "treatment" }, { "unitId": "user_43", "bucket": 1211, "groupBucket": 9800, "inExperiment": true, "group": "treatment" } ], "algorithm": "sha256(salt + '.' + unitId), first 8 bytes big-endian uint64 mod 10000", "sources": [ { "source": "NIST FIPS 180-4 Secure Hash Standard (SHA-256)", "url": "https://csrc.nist.gov/pubs/fips/180-4/upd1/final", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Group assignment hashes salt + '.group.' + unitId so changing trafficShare does not reshuffle groups. Changing a salt r…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/flag-rule-evaluate $0.01 USDC. Evaluate ordered feature-flag rules (attribute operators, semver comparison, deterministic percentage rollout) for one user context and explain which rule matched. Inputs: flag, context Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-flag-rule-evaluate Example request: ```json { "flag": { "key": "new_checkout", "salt": "new_checkout", "default": false, "rules": [ { "id": "staff", "conditions": [ { "attribute": "email", "op": "ends_with", "value": "@acme.com" } ], "serve": true }, { "id": "uk_rollout", "conditions": [ { "attribute": "country", "op": "in", "value": [ "GB" ] } ], "rollout": 0.25, "serve": true } ] }, "context": { "userId": "user_42", "email": "a@b.com", "country": "GB", "appVersion": "5.2.0" } } ``` Example response: ```json { "value": false, "matchedRule": null, "trace": [ { "rule": "staff", "matched": false, "failedCondition": 0 }, { "rule": "uk_rollout", "matched": true, "inRollout": false, "bucket": 3933 } ], "reason": "default", "sources": [ { "source": "NIST FIPS 180-4 Secure Hash Standard (SHA-256)", "url": "https://csrc.nist.gov/pubs/fips/180-4/upd1/final", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "Semantic Versioning 2.0.0", "url": "https://semver.org/", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Regex conditions run on the JavaScript engine with a guard (no backreferences, lookaround or nested quantifiers, 200 ch…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/novelty-effect-detect $0.01 USDC. OLS trend test on daily lift with early-versus-late means, classifying the pattern as novelty, primacy or stable. Inputs: daily, liftType?, earlyDays?, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-novelty-effect-detect Example request: ```json { "daily": [ { "day": 1, "lift": 0.12 }, { "day": 2, "lift": 0.1 }, { "day": 3, "lift": 0.09 }, { "day": 4, "lift": 0.07 }, { "day": 5, "lift": 0.06 }, { "day": 6, "lift": 0.05 }, { "day": 7, "lift": 0.05 }, { "day": 8, "lift": 0.04 }, { "day": 9, "lift": 0.04 }, { "day": 10, "lift": 0.04 }, { "day": 11, "lift": 0.03 }, { "day": 12, "lift": 0.04 }, { "day": 13, "lift": 0.03 }, { "day": 14, "lift": 0.03 } ], "liftType": "relative", "earlyDays": 7, "alpha": 0.05 } ``` Example response: ```json { "slopePerDay": -0.006175824176, "slopeSe": 0.000821950598, "t": -7.513619664, "pValue": 0.000007103095267, "earlyMean": 0.07714285714, "lateMean": 0.03571428571, "earlyDays": 7, "liftType": "relative", "pattern": "novelty", "sources": [ { "source": "NIST/SEMATECH e-Handbook: linear least squares regression", "url": "https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd431.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "A straight-line trend is a coarse model: weekly seasonality, holidays and campaigns can mimic or hide a novelty effect.…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ``` ### POST /api/v1/experimentation/segment-lift $0.02 USDC. Per-segment conversion lift with Benjamini-Hochberg adjusted p-values, a Cochran Q heterogeneity test and suppression of small segments. Inputs: segments, minUsersPerArm?, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-segment-lift Example request: ```json { "segments": [ { "name": "mobile", "control": { "users": 5000, "conversions": 500 }, "treatment": { "users": 5000, "conversions": 600 } }, { "name": "desktop", "control": { "users": 4000, "conversions": 480 }, "treatment": { "users": 4000, "conversions": 484 } }, { "name": "tablet", "control": { "users": 1000, "conversions": 100 }, "treatment": { "users": 1000, "conversions": 110 } } ], "minUsersPerArm": 100, "alpha": 0.05 } ``` Example response: ```json { "segments": [ { "name": "mobile", "absoluteLift": 0.02, "ci": [ 0.007741212578, 0.03225878742 ], "z": 3.197647397, "pValue": 0.001385535823, "pAdjustedBH": 0.004156607469, "significant": true }, { "name": "desktop", "absoluteLift": 0.001, "ci": [ -0.01326737336, 0.01526737336 ], "z": 0.1373738484, "pValue": 0.890735301, "pAdjustedBH": 0.890735301, "significant": false }, "…" ], "heterogeneity": { "pooledLift": 0.01172400678, "Q": 3.936950198, "df": 2, "pValue": 0.1396696763, "I2": 0.4919925579 }, "suppressed": [], "sources": [ { "source": "Benjamini & Hochberg (1995) Controlling the false discovery rate", "url": "https://doi.org/10.1111/j.2517-6161.1995.tb02031.x", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" }, { "source": "Higgins & Thompson (2002) Quantifying heterogeneity in a meta-analysis", "url": "https://doi.org/10.1002/sim.1186", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Segments are only as valid as randomisation within each segment; segments chosen after seeing results inflate false pos…", "Heterogeneity uses inverse-variance weights on absolute lift.", "…" ] } ``` ### POST /api/v1/experimentation/guardrail-check $0.02 USDC. One-sided non-inferiority tests of guardrail metrics against caller-supplied relative or absolute margins, with bounds and an overall pass flag. Inputs: alpha?, metrics Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-guardrail-check Example request: ```json { "alpha": 0.05, "metrics": [ { "name": "latency_ms", "type": "mean", "higherIsBetter": false, "marginType": "relative", "margin": 0.02, "control": { "n": 5000, "mean": 200, "sd": 80 }, "treatment": { "n": 5000, "mean": 201.5, "sd": 82 } } ] } ``` Example response: ```json { "metrics": [ { "name": "latency_ms", "difference": 1.5, "se": 1.620123452, "margin": 4, "z": 1.543092285, "pValue": 0.06140419316, "upperBound": 4.164865936, "pass": false } ], "allPass": false, "sources": [ { "source": "Walker & Nowacki (2011) Understanding equivalence and noninferiority testing", "url": "https://doi.org/10.1007/s11606-010-1513-8", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "One-sided non-inferiority z-test at the caller's alpha with an unpooled standard error; the margin is the caller's busi…", "Failing to show non-inferiority is not evidence of harm, and passing does not show no change.", "…" ] } ``` ### POST /api/v1/experimentation/holdout-impact $0.01 USDC. Exposed-versus-holdout impact with confidence interval and p-value, plus diluted population-level impact for partial exposure. Inputs: metricType, holdout, exposed, exposureRate?, alpha? Docs: https://www.brainiac.software/api/experimentation/docs#experimentation-holdout-impact Example request: ```json { "metricType": "proportion", "holdout": { "users": 20000, "conversions": 2000 }, "exposed": { "users": 180000, "conversions": 18720 }, "exposureRate": 0.6, "alpha": 0.05 } ``` Example response: ```json { "absoluteImpact": 0.004, "ci": [ -0.0003903582134, 0.008390358213 ], "relativeImpact": 0.04, "pValue": 0.07414809347, "dilutedImpact": 0.0024, "dilutedCi": [ -0.000234214928, 0.005034214928 ], "significant": false, "sources": [ { "source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)", "url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm", "retrievedAt": "2026-11-14T00:00:00.000Z", "status": "ok" } ], "limitations": [ "Diluted impact scales the exposed-vs-holdout difference by exposureRate (intent-to-treat style); it assumes the unexpos…", "Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…" ] } ```