https://www.brainiac.software/api/v1/experimentation/ab-test-frequentist$0.02 USDCRun a frequentist A/B test scorecard
Two-arm A/B scorecard for conversion (two-proportion z) or continuous (Welch t) metrics: absolute and relative lift, confidence intervals, p-value and significance. Pure computation on supplied aggregates.
Body
| Field | Required | Notes |
|---|---|---|
metricType | Yes | string; One of: proportion, mean |
alpha | No | number; Default: 0.05 |
alternative | No | string; One of: two-sided, greater, less; Default: "two-sided" |
control | Yes | — |
treatment | Yes | — |
Request exampleJSON
{
"metricType": "proportion",
"alpha": 0.05,
"alternative": "two-sided",
"control": {
"users": 10000,
"conversions": 1000
},
"treatment": {
"users": 10000,
"conversions": 1100
}
}Response exampleJSON
{
"controlRate": 0.1,
"treatmentRate": 0.11,
"absoluteLift": 0.01,
"absoluteCi": [
0.001504059132,
0.01849594087
],
"relativeLift": 0.1,
"relativeCi": [
0.01087011253,
0.1891298875
],
"statistic": 2.306634774,
"pValue": 0.02107518927,
"significant": true,
"test": "two-proportion z (pooled SE for test, unpooled SE for CI)",
"sources": [
{
"source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)",
"url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "Deng, Knoblich, Lu (2018) Applying the Delta Method in Metric Analytics",
"url": "https://arxiv.org/abs/1803.06336",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Normal approximation: unreliable when expected successes or failures per arm are below about 10. Relative-lift interval…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/ab-test-frequentist" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"metricType": "proportion",
"alpha": 0.05,
"alternative": "two-sided",
"control": {
"users": 10000,
"conversions": 1000
},
"treatment": {
"users": 10000,
"conversions": 1100
}
}
JSONhttps://www.brainiac.software/api/v1/experimentation/bayesian-ab$0.03 USDCCompare A/B arms with a Bayesian Beta-Binomial model
Posterior, probability to beat control, probability of being best, expected loss and equal-tailed credible intervals for 2-10 conversion arms. Deterministic: exact Miller sum or quadrature, no Monte Carlo.
Body
| Field | Required | Notes |
|---|---|---|
prior | No | object; Default: {"alpha":1,"beta":1}; Object |
arms | Yes | array; Array |
credibleLevel | No | number; Default: 0.95 |
Request exampleJSON
{
"prior": {
"alpha": 1,
"beta": 1
},
"arms": [
{
"name": "A",
"users": 10000,
"conversions": 1000
},
{
"name": "B",
"users": 10000,
"conversions": 1100
}
],
"credibleLevel": 0.95
}Response exampleJSON
{
"arms": [
{
"name": "A",
"posterior": {
"alpha": 1001,
"beta": 9001
},
"mean": 0.100079984,
"credible": [
0.09427511307,
0.106036332
],
"probBeatControl": null,
"probBest": 0.01054264559,
"expectedLoss": 0.01001358271
},
{
"name": "B",
"posterior": {
"alpha": 1101,
"beta": 8901
},
"mean": 0.1100779844,
"credible": [
0.1040187816,
0.1162848782
],
"probBeatControl": 0.9894573544,
"probBest": 0.9894573544,
"expectedLoss": 0.00001558231261
}
],
"control": "A",
"credibleLevel": 0.95,
"method": {
"probBeatControl": "exact (Miller 2015)",
"probBest": "pairwise probability to beat control",
"expectedLoss": "deterministic Simpson quadrature of E[max_j p_j] minus posterior mean"
},
"sources": [
{
"source": "Miller (2015) Formulas for Bayesian A/B Testing",
"url": "https://www.evanmiller.org/bayesian-ab-testing.html",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Independent Beta-Binomial posteriors; no Monte Carlo, so results are reproducible. Quadrature spans the 1e-9 to 1-1e-9 …",
"Probabilities describe the posterior under the supplied prior, not the long-run error rate of a procedure.",
"…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/bayesian-ab" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"prior": {
"alpha": 1,
"beta": 1
},
"arms": [
{
"name": "A",
"users": 10000,
"conversions": 1000
},
{
"name": "B",
"users": 10000,
"conversions": 1100
}
],
"credibleLevel": 0.95
}
JSONhttps://www.brainiac.software/api/v1/experimentation/sequential-test$0.04 USDCCompute sequential testing boundaries or an always-valid p-value
Lan-DeMets group-sequential boundaries (O'Brien-Fleming or Pocock spending) for planned looks with a stop/continue decision, or an always-valid mSPRT p-value for continuous monitoring.
Body
| Field | Required | Notes |
|---|---|---|
mode | Yes | string; One of: group_sequential, msprt |
alpha | No | number; Default: 0.05 |
spending | No | string; One of: obrien_fleming, pocock; Default: "obrien_fleming" |
informationFractions | No | array; Array |
observedZ | No | array; Array |
estimate | No | number |
variance | No | number |
tau2 | No | number |
Request exampleJSON
{
"mode": "group_sequential",
"alpha": 0.05,
"spending": "obrien_fleming",
"informationFractions": [
0.2,
0.4,
0.6,
0.8,
1
],
"observedZ": [
1.2,
2.1,
null,
null,
null
]
}Response exampleJSON
{
"boundaries": [
4.876884949,
3.357011922,
"…"
],
"cumulativeAlpha": [
0.000001077742526,
0.0007883035134,
"…"
],
"decision": {
"look": 2,
"crossed": false,
"boundary": 3.357011922,
"observedZ": 2.1
},
"spending": "obrien_fleming",
"sources": [
{
"source": "Lan & DeMets (1983) Discrete sequential boundaries for clinical trials, Biometrika 70(3)",
"url": "https://doi.org/10.1093/biomet/70.3.659",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Two-sided symmetric boundaries; the total type I error is alpha and each side spends alpha/2. Boundaries assume the pla…",
"Numerical recursion on a Simpson grid (at least 2,001 points) with bisection to 1e-9; agrees with published Lan-DeMets …",
"…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/sequential-test" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"mode": "group_sequential",
"alpha": 0.05,
"spending": "obrien_fleming",
"informationFractions": [
0.2,
0.4,
0.6,
0.8,
1
],
"observedZ": [
1.2,
2.1,
null,
null,
null
]
}
JSONhttps://www.brainiac.software/api/v1/experimentation/cuped-adjust$0.03 USDCApply CUPED variance reduction and test the adjusted effect
CUPED adjustment with a pre-experiment covariate on unit-level data: theta, variance reduction, and Welch tests on raw and adjusted values.
Body
| Field | Required | Notes |
|---|---|---|
units | Yes | array; Array |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"units": [
{
"group": "control",
"y": 11,
"x": 10
},
{
"group": "treatment",
"y": 13,
"x": 12
},
{
"group": "control",
"y": 9,
"x": 9
},
{
"group": "treatment",
"y": 17,
"x": 15
},
{
"group": "control",
"y": 12,
"x": 11
},
{
"group": "treatment",
"y": 15,
"x": 14
},
{
"group": "control",
"y": 9,
"x": 8
},
{
"group": "treatment",
"y": 14,
"x": 13
},
{
"group": "control",
"y": 10,
"x": 10
},
{
"group": "treatment",
"y": 14,
"x": 12
}
],
"alpha": 0.05
}Response exampleJSON
{
"theta": 1.18018018,
"varianceReduction": 0.9602708298,
"raw": {
"diff": 4.4,
"se": 0.894427191,
"pValue": 0.001242936514,
"ci": [
2.32934137,
6.47065863
],
"significant": true
},
"adjusted": {
"controlMean": 12.32432432,
"treatmentMean": 12.47567568,
"diff": 0.1513513514,
"se": 0.3536446295,
"pValue": 0.679972045,
"ci": [
-0.6643475398,
0.9670502425
],
"significant": false
},
"groups": {
"control": "control",
"treatment": "treatment"
},
"unitsWithoutCovariate": 0,
"sources": [
{
"source": "Deng, Xu, Kohavi, Walker (2013) CUPED: improving sensitivity of controlled experiments",
"url": "https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)",
"url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"theta is estimated on both groups pooled, as in the CUPED paper; the covariate must be measured before assignment or it…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/cuped-adjust" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"units": [
{
"group": "control",
"y": 11,
"x": 10
},
{
"group": "treatment",
"y": 13,
"x": 12
},
{
"group": "control",
"y": 9,
"x": 9
},
{
"group": "treatment",
"y": 17,
"x": 15
},
{
"group": "control",
"y": 12,
"x": 11
},
{
"group": "treatment",
"y": 15,
"x": 14
},
{
"group": "control",
"y": 9,
"x": 8
},
{
"group": "treatment",
"y": 14,
"x": 13
},
{
"group": "control",
"y": 10,
"x": 10
},
{
"group": "treatment",
"y": 14,
"x": 12
}
],
"alpha": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/sample-ratio-mismatch$0.01 USDCDetect A/B sample ratio mismatch
Chi-square goodness-of-fit test of observed arm counts against expected traffic shares, with an exact binomial p-value for tiny two-arm counts and generic causes when a mismatch is flagged.
Body
| Field | Required | Notes |
|---|---|---|
arms | Yes | array; Array |
threshold | No | number; Default: 0.0005 |
Request exampleJSON
{
"arms": [
{
"name": "control",
"observed": 50000,
"expectedShare": 0.5
},
{
"name": "treatment",
"observed": 49000,
"expectedShare": 0.5
}
],
"threshold": 0.0005
}Response exampleJSON
{
"chiSquare": 10.1010101,
"df": 1,
"pValue": 0.001481880775,
"srm": false,
"threshold": 0.0005,
"arms": [
{
"name": "control",
"observed": 50000,
"expected": 49500,
"deviationPct": 1.01
},
{
"name": "treatment",
"observed": 49000,
"expected": 49500,
"deviationPct": -1.01
}
],
"hints": [],
"sources": [
{
"source": "NIST/SEMATECH e-Handbook: chi-square goodness-of-fit test",
"url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"A non-significant result does not prove assignment is balanced; the default threshold 0.0005 is conservative on purpose.",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/sample-ratio-mismatch" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"arms": [
{
"name": "control",
"observed": 50000,
"expectedShare": 0.5
},
{
"name": "treatment",
"observed": 49000,
"expectedShare": 0.5
}
],
"threshold": 0.0005
}
JSONhttps://www.brainiac.software/api/v1/experimentation/sample-size-mde$0.01 USDCCalculate A/B sample size or minimum detectable effect
Required users per arm and duration for a target minimum detectable effect, or the MDE for a fixed sample, for proportion or mean metrics with power, multi-arm Bonferroni and unequal allocation.
Body
| Field | Required | Notes |
|---|---|---|
metricType | Yes | string; One of: proportion, mean |
baseline | No | number |
baselineSd | No | number |
baselineMean | No | number |
mde | No | number |
mdeType | No | string; One of: relative, absolute; Default: "relative" |
nPerArm | No | integer |
alpha | No | number; Default: 0.05 |
alternative | No | string; One of: two-sided, greater, less; Default: "two-sided" |
power | No | number; Default: 0.8 |
arms | No | integer; Default: 2 |
allocation | No | array; Array |
dailyUsers | No | number |
correction | No | string; One of: bonferroni, none; Default: "none" |
Request exampleJSON
{
"metricType": "proportion",
"baseline": 0.1,
"mde": 0.1,
"mdeType": "relative",
"alpha": 0.05,
"power": 0.8,
"arms": 2,
"allocation": [
0.5,
0.5
],
"dailyUsers": 2000,
"correction": "bonferroni"
}Response exampleJSON
{
"nPerArm": 14749,
"nByArm": [
14749,
14749
],
"totalN": 29498,
"days": 15,
"alphaPerComparison": 0.05,
"formula": "unpooled normal approximation",
"sources": [
{
"source": "NIST/SEMATECH e-Handbook: sample sizes required for means",
"url": "https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Normal approximation; for small samples or rare events check the result by simulation.",
"Allocation index 0 is the control; every other arm is compared with it.",
"…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/sample-size-mde" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"metricType": "proportion",
"baseline": 0.1,
"mde": 0.1,
"mdeType": "relative",
"alpha": 0.05,
"power": 0.8,
"arms": 2,
"allocation": [
0.5,
0.5
],
"dailyUsers": 2000,
"correction": "bonferroni"
}
JSONhttps://www.brainiac.software/api/v1/experimentation/multiple-testing-correct$0.01 USDCAdjust p-values for multiple testing
Bonferroni, Holm and Benjamini-Hochberg adjusted p-values and reject decisions across metrics or arms, returned in input order.
Body
| Field | Required | Notes |
|---|---|---|
pValues | Yes | array; Array |
methods | No | array; Default: ["bonferroni","holm","benjamini_hochberg"]; Array |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"pValues": [
{
"id": "m1",
"p": 0.01
},
{
"id": "m2",
"p": 0.04
},
{
"id": "m3",
"p": 0.03
},
{
"id": "m4",
"p": 0.005
}
],
"methods": [
"bonferroni",
"holm",
"benjamini_hochberg"
],
"alpha": 0.05
}Response exampleJSON
{
"results": [
{
"id": "m1",
"p": 0.01,
"bonferroni": 0.04,
"holm": 0.03,
"benjamini_hochberg": 0.02,
"rejected": {
"bonferroni": true,
"holm": true,
"benjamini_hochberg": true
}
},
{
"id": "m2",
"p": 0.04,
"bonferroni": 0.16,
"holm": 0.06,
"benjamini_hochberg": 0.04,
"rejected": {
"bonferroni": false,
"holm": false,
"benjamini_hochberg": true
}
},
"…"
],
"alpha": 0.05,
"sources": [
{
"source": "Holm (1979) A simple sequentially rejective multiple test procedure",
"url": "https://www.jstor.org/stable/4615733",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "Benjamini & Hochberg (1995) Controlling the false discovery rate",
"url": "https://doi.org/10.1111/j.2517-6161.1995.tb02031.x",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Bonferroni and Holm control the family-wise error rate; Benjamini-Hochberg controls the false discovery rate and assume…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/multiple-testing-correct" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"pValues": [
{
"id": "m1",
"p": 0.01
},
{
"id": "m2",
"p": 0.04
},
{
"id": "m3",
"p": 0.03
},
{
"id": "m4",
"p": 0.005
}
],
"methods": [
"bonferroni",
"holm",
"benjamini_hochberg"
],
"alpha": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/delta-method-ratio$0.03 USDCEstimate a ratio metric with the delta method
Confidence interval for a ratio metric whose analysis unit differs from the randomisation unit, plus the treatment-vs-control difference and relative lift. Accepts unit rows or aggregates.
Body
| Field | Required | Notes |
|---|---|---|
arms | Yes | array; Array |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"arms": [
{
"name": "control",
"units": [
{
"num": 10,
"den": 2
},
{
"num": 0,
"den": 1
},
{
"num": 5,
"den": 1
},
{
"num": 20,
"den": 4
},
{
"num": 3,
"den": 1
},
{
"num": 0,
"den": 2
},
{
"num": 8,
"den": 2
},
{
"num": 12,
"den": 3
}
]
}
],
"alpha": 0.05
}Response exampleJSON
{
"arms": [
{
"name": "control",
"ratio": 3.625,
"se": 0.6931585317,
"ci": [
2.266434242,
4.983565758
]
}
],
"difference": {
"estimate": null,
"se": null,
"ci": null,
"pValue": null
},
"relativeLift": null,
"relativeCi": null,
"sources": [
{
"source": "Deng, Knoblich, Lu (2018) Applying the Delta Method in Metric Analytics",
"url": "https://arxiv.org/abs/1803.06336",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"First-order delta method with a normal quantile; it is accurate for large numbers of randomisation units and unreliable…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/delta-method-ratio" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"arms": [
{
"name": "control",
"units": [
{
"num": 10,
"den": 2
},
{
"num": 0,
"den": 1
},
{
"num": 5,
"den": 1
},
{
"num": 20,
"den": 4
},
{
"num": 3,
"den": 1
},
{
"num": 0,
"den": 2
},
{
"num": 8,
"den": 2
},
{
"num": 12,
"den": 3
}
]
}
],
"alpha": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/bandit-allocate$0.03 USDCCompute multi-armed bandit traffic weights
Next-period traffic weights from arm outcomes: Thompson-sampling probabilities by deterministic quadrature, or UCB1 indices, with an exploration floor.
Body
| Field | Required | Notes |
|---|---|---|
policy | Yes | string; One of: thompson, ucb1 |
arms | Yes | array; Array |
prior | No | object; Default: {"alpha":1,"beta":1}; Object |
minShare | No | number; Default: 0 |
Request exampleJSON
{
"policy": "thompson",
"arms": [
{
"name": "A",
"successes": 100,
"failures": 900
},
{
"name": "B",
"successes": 120,
"failures": 880
},
{
"name": "C",
"successes": 90,
"failures": 910
}
],
"prior": {
"alpha": 1,
"beta": 1
},
"minShare": 0.05
}Response exampleJSON
{
"policy": "thompson",
"weights": [
{
"name": "A",
"probBest": 0.07474923213,
"weight": 0.07176986341
},
{
"name": "B",
"probBest": 0.9146879375,
"weight": 0.8782301366
},
"…"
],
"sources": [
{
"source": "Russo et al. (2018) A Tutorial on Thompson Sampling",
"url": "https://arxiv.org/abs/1707.02038",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "Miller (2015) Formulas for Bayesian A/B Testing",
"url": "https://www.evanmiller.org/bayesian-ab-testing.html",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Thompson weights are exact posterior probabilities of being best (quadrature), not random draws, so repeated calls retu…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/bandit-allocate" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"policy": "thompson",
"arms": [
{
"name": "A",
"successes": 100,
"failures": 900
},
{
"name": "B",
"successes": 120,
"failures": 880
},
{
"name": "C",
"successes": 90,
"failures": 910
}
],
"prior": {
"alpha": 1,
"beta": 1
},
"minShare": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/bucket-assign$0.01 USDCAssign units to experiment groups deterministically
Stateless SHA-256 bucketing of unit ids into an experiment with traffic gating, ordered group shares and an optional layer, documented so any SDK can reproduce it.
Body
| Field | Required | Notes |
|---|---|---|
unitIds | Yes | array; Array |
experiment | Yes | object; Object |
layer | No | — |
Request exampleJSON
{
"unitIds": [
"user_42",
"user_43"
],
"experiment": {
"salt": "exp_checkout_v2",
"trafficShare": 1,
"groups": [
{
"name": "control",
"share": 0.5
},
{
"name": "treatment",
"share": 0.5
}
]
},
"layer": null
}Response exampleJSON
{
"assignments": [
{
"unitId": "user_42",
"bucket": 7765,
"groupBucket": 8754,
"inExperiment": true,
"group": "treatment"
},
{
"unitId": "user_43",
"bucket": 1211,
"groupBucket": 9800,
"inExperiment": true,
"group": "treatment"
}
],
"algorithm": "sha256(salt + '.' + unitId), first 8 bytes big-endian uint64 mod 10000",
"sources": [
{
"source": "NIST FIPS 180-4 Secure Hash Standard (SHA-256)",
"url": "https://csrc.nist.gov/pubs/fips/180-4/upd1/final",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Group assignment hashes salt + '.group.' + unitId so changing trafficShare does not reshuffle groups. Changing a salt r…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/bucket-assign" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"unitIds": [
"user_42",
"user_43"
],
"experiment": {
"salt": "exp_checkout_v2",
"trafficShare": 1,
"groups": [
{
"name": "control",
"share": 0.5
},
{
"name": "treatment",
"share": 0.5
}
]
},
"layer": null
}
JSONhttps://www.brainiac.software/api/v1/experimentation/flag-rule-evaluate$0.01 USDCEvaluate a feature-flag rule set
Evaluate ordered feature-flag rules (attribute operators, semver comparison, deterministic percentage rollout) for one user context and explain which rule matched.
Body
| Field | Required | Notes |
|---|---|---|
flag | Yes | object; Object |
context | Yes | object |
Request exampleJSON
{
"flag": {
"key": "new_checkout",
"salt": "new_checkout",
"default": false,
"rules": [
{
"id": "staff",
"conditions": [
{
"attribute": "email",
"op": "ends_with",
"value": "@acme.com"
}
],
"serve": true
},
{
"id": "uk_rollout",
"conditions": [
{
"attribute": "country",
"op": "in",
"value": [
"GB"
]
}
],
"rollout": 0.25,
"serve": true
}
]
},
"context": {
"userId": "user_42",
"email": "a@b.com",
"country": "GB",
"appVersion": "5.2.0"
}
}Response exampleJSON
{
"value": false,
"matchedRule": null,
"trace": [
{
"rule": "staff",
"matched": false,
"failedCondition": 0
},
{
"rule": "uk_rollout",
"matched": true,
"inRollout": false,
"bucket": 3933
}
],
"reason": "default",
"sources": [
{
"source": "NIST FIPS 180-4 Secure Hash Standard (SHA-256)",
"url": "https://csrc.nist.gov/pubs/fips/180-4/upd1/final",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "Semantic Versioning 2.0.0",
"url": "https://semver.org/",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Regex conditions run on the JavaScript engine with a guard (no backreferences, lookaround or nested quantifiers, 200 ch…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/flag-rule-evaluate" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"flag": {
"key": "new_checkout",
"salt": "new_checkout",
"default": false,
"rules": [
{
"id": "staff",
"conditions": [
{
"attribute": "email",
"op": "ends_with",
"value": "@acme.com"
}
],
"serve": true
},
{
"id": "uk_rollout",
"conditions": [
{
"attribute": "country",
"op": "in",
"value": [
"GB"
]
}
],
"rollout": 0.25,
"serve": true
}
]
},
"context": {
"userId": "user_42",
"email": "a@b.com",
"country": "GB",
"appVersion": "5.2.0"
}
}
JSONhttps://www.brainiac.software/api/v1/experimentation/novelty-effect-detect$0.01 USDCDetect novelty or primacy effects in daily lift
OLS trend test on daily lift with early-versus-late means, classifying the pattern as novelty, primacy or stable.
Body
| Field | Required | Notes |
|---|---|---|
daily | Yes | array; Array |
liftType | No | string; One of: relative, absolute |
earlyDays | No | integer |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"daily": [
{
"day": 1,
"lift": 0.12
},
{
"day": 2,
"lift": 0.1
},
{
"day": 3,
"lift": 0.09
},
{
"day": 4,
"lift": 0.07
},
{
"day": 5,
"lift": 0.06
},
{
"day": 6,
"lift": 0.05
},
{
"day": 7,
"lift": 0.05
},
{
"day": 8,
"lift": 0.04
},
{
"day": 9,
"lift": 0.04
},
{
"day": 10,
"lift": 0.04
},
{
"day": 11,
"lift": 0.03
},
{
"day": 12,
"lift": 0.04
},
{
"day": 13,
"lift": 0.03
},
{
"day": 14,
"lift": 0.03
}
],
"liftType": "relative",
"earlyDays": 7,
"alpha": 0.05
}Response exampleJSON
{
"slopePerDay": -0.006175824176,
"slopeSe": 0.000821950598,
"t": -7.513619664,
"pValue": 0.000007103095267,
"earlyMean": 0.07714285714,
"lateMean": 0.03571428571,
"earlyDays": 7,
"liftType": "relative",
"pattern": "novelty",
"sources": [
{
"source": "NIST/SEMATECH e-Handbook: linear least squares regression",
"url": "https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd431.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"A straight-line trend is a coarse model: weekly seasonality, holidays and campaigns can mimic or hide a novelty effect.…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/novelty-effect-detect" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"daily": [
{
"day": 1,
"lift": 0.12
},
{
"day": 2,
"lift": 0.1
},
{
"day": 3,
"lift": 0.09
},
{
"day": 4,
"lift": 0.07
},
{
"day": 5,
"lift": 0.06
},
{
"day": 6,
"lift": 0.05
},
{
"day": 7,
"lift": 0.05
},
{
"day": 8,
"lift": 0.04
},
{
"day": 9,
"lift": 0.04
},
{
"day": 10,
"lift": 0.04
},
{
"day": 11,
"lift": 0.03
},
{
"day": 12,
"lift": 0.04
},
{
"day": 13,
"lift": 0.03
},
{
"day": 14,
"lift": 0.03
}
],
"liftType": "relative",
"earlyDays": 7,
"alpha": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/segment-lift$0.02 USDCMeasure A/B lift per segment with FDR control
Per-segment conversion lift with Benjamini-Hochberg adjusted p-values, a Cochran Q heterogeneity test and suppression of small segments.
Body
| Field | Required | Notes |
|---|---|---|
segments | Yes | array; Array |
minUsersPerArm | No | integer; Default: 100 |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"segments": [
{
"name": "mobile",
"control": {
"users": 5000,
"conversions": 500
},
"treatment": {
"users": 5000,
"conversions": 600
}
},
{
"name": "desktop",
"control": {
"users": 4000,
"conversions": 480
},
"treatment": {
"users": 4000,
"conversions": 484
}
},
{
"name": "tablet",
"control": {
"users": 1000,
"conversions": 100
},
"treatment": {
"users": 1000,
"conversions": 110
}
}
],
"minUsersPerArm": 100,
"alpha": 0.05
}Response exampleJSON
{
"segments": [
{
"name": "mobile",
"absoluteLift": 0.02,
"ci": [
0.007741212578,
0.03225878742
],
"z": 3.197647397,
"pValue": 0.001385535823,
"pAdjustedBH": 0.004156607469,
"significant": true
},
{
"name": "desktop",
"absoluteLift": 0.001,
"ci": [
-0.01326737336,
0.01526737336
],
"z": 0.1373738484,
"pValue": 0.890735301,
"pAdjustedBH": 0.890735301,
"significant": false
},
"…"
],
"heterogeneity": {
"pooledLift": 0.01172400678,
"Q": 3.936950198,
"df": 2,
"pValue": 0.1396696763,
"I2": 0.4919925579
},
"suppressed": [],
"sources": [
{
"source": "Benjamini & Hochberg (1995) Controlling the false discovery rate",
"url": "https://doi.org/10.1111/j.2517-6161.1995.tb02031.x",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
},
{
"source": "Higgins & Thompson (2002) Quantifying heterogeneity in a meta-analysis",
"url": "https://doi.org/10.1002/sim.1186",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Segments are only as valid as randomisation within each segment; segments chosen after seeing results inflate false pos…",
"Heterogeneity uses inverse-variance weights on absolute lift.",
"…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/segment-lift" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"segments": [
{
"name": "mobile",
"control": {
"users": 5000,
"conversions": 500
},
"treatment": {
"users": 5000,
"conversions": 600
}
},
{
"name": "desktop",
"control": {
"users": 4000,
"conversions": 480
},
"treatment": {
"users": 4000,
"conversions": 484
}
},
{
"name": "tablet",
"control": {
"users": 1000,
"conversions": 100
},
"treatment": {
"users": 1000,
"conversions": 110
}
}
],
"minUsersPerArm": 100,
"alpha": 0.05
}
JSONhttps://www.brainiac.software/api/v1/experimentation/guardrail-check$0.02 USDCCheck A/B guardrail metrics for non-inferiority
One-sided non-inferiority tests of guardrail metrics against caller-supplied relative or absolute margins, with bounds and an overall pass flag.
Body
| Field | Required | Notes |
|---|---|---|
alpha | No | number; Default: 0.05 |
metrics | Yes | array; Array |
Request exampleJSON
{
"alpha": 0.05,
"metrics": [
{
"name": "latency_ms",
"type": "mean",
"higherIsBetter": false,
"marginType": "relative",
"margin": 0.02,
"control": {
"n": 5000,
"mean": 200,
"sd": 80
},
"treatment": {
"n": 5000,
"mean": 201.5,
"sd": 82
}
}
]
}Response exampleJSON
{
"metrics": [
{
"name": "latency_ms",
"difference": 1.5,
"se": 1.620123452,
"margin": 4,
"z": 1.543092285,
"pValue": 0.06140419316,
"upperBound": 4.164865936,
"pass": false
}
],
"allPass": false,
"sources": [
{
"source": "Walker & Nowacki (2011) Understanding equivalence and noninferiority testing",
"url": "https://doi.org/10.1007/s11606-010-1513-8",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"One-sided non-inferiority z-test at the caller's alpha with an unpooled standard error; the margin is the caller's busi…",
"Failing to show non-inferiority is not evidence of harm, and passing does not show no change.",
"…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/guardrail-check" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"alpha": 0.05,
"metrics": [
{
"name": "latency_ms",
"type": "mean",
"higherIsBetter": false,
"marginType": "relative",
"margin": 0.02,
"control": {
"n": 5000,
"mean": 200,
"sd": 80
},
"treatment": {
"n": 5000,
"mean": 201.5,
"sd": 82
}
}
]
}
JSONhttps://www.brainiac.software/api/v1/experimentation/holdout-impact$0.01 USDCMeasure cumulative impact against a long-term holdout
Exposed-versus-holdout impact with confidence interval and p-value, plus diluted population-level impact for partial exposure.
Body
| Field | Required | Notes |
|---|---|---|
metricType | Yes | string; One of: proportion, mean |
holdout | Yes | — |
exposed | Yes | — |
exposureRate | No | number |
alpha | No | number; Default: 0.05 |
Request exampleJSON
{
"metricType": "proportion",
"holdout": {
"users": 20000,
"conversions": 2000
},
"exposed": {
"users": 180000,
"conversions": 18720
},
"exposureRate": 0.6,
"alpha": 0.05
}Response exampleJSON
{
"absoluteImpact": 0.004,
"ci": [
-0.0003903582134,
0.008390358213
],
"relativeImpact": 0.04,
"pValue": 0.07414809347,
"dilutedImpact": 0.0024,
"dilutedCi": [
-0.000234214928,
0.005034214928
],
"significant": false,
"sources": [
{
"source": "NIST/SEMATECH e-Handbook: two-sample t-test (Welch)",
"url": "https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm",
"retrievedAt": "2026-11-14T00:00:00.000Z",
"status": "ok"
}
],
"limitations": [
"Diluted impact scales the exposed-vs-holdout difference by exposureRate (intent-to-treat style); it assumes the unexpos…",
"Pure computation on the supplied input: no events are stored and no network is called. Results are statistical calculat…"
]
}cURL
curl -i -X POST "https://www.brainiac.software/api/v1/experimentation/holdout-impact" \
-H 'content-type: application/json' \
-H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
--data-binary @- <<'JSON'
{
"metricType": "proportion",
"holdout": {
"users": 20000,
"conversions": 2000
},
"exposed": {
"users": 180000,
"conversions": 18720
},
"exposureRate": 0.6,
"alpha": 0.05
}
JSON