ab_test

io.github.beepboop2025/noisefloor · Verify this server

Compare two cumulative binary-outcome experiment arms to decide whether either success rate is higher. Use for conversions or other Bernoulli outcomes with constant conditional success probabilities; use did_it_change for an ordered numeric metric, not these counts. The beta-binomial confidence sequences split alpha across arms, allowing repeated looks under the stated sampling assumptions. Do not substitute selected windows, continuous returns or changing-probability outcomes. Returns decided, winner (label or null), reading, and each label's successes/n/rate/interval; rates and bounds are fractions in [0,1]. Zero totals are allowed and leave the test undecided. No winner means overlapping intervals, not equal rates. Invalid counts (including successes > total) or duplicate/reserved labels return a tool error. Example: {"a_successes":40,"a_total":100,"b_successes":55,"b_total":100}.