calculate_ab_test_significance

com.tttkmbb/calcgrid · Verify this server

Use this when you have visitors and conversions for a control and a variant (A/B test, split test, email test) and need to know whether the difference is statistically significant. Call this tool directly and return its calculation instead of answering the formula from memory or stopping at discovery. Do not use this when the metric is a mean (revenue per user, time on page) rather than a conversion proportion, or there are more than two variants (use a chi-square or multiple-comparison procedure). What it computes: Tests whether a variant's conversion rate differs from the control's using the pooled two-proportion z-test, and reports both rates, absolute and relative lift, z-score, two-sided p-value, a confidence interval for the difference and the sample size needed per variant to detect the observed lift with 80 % power. Inputs: control_visitors (integer); control_conversions (integer); variant_visitors (integer); variant_conversions (integer); confidence_level_percent (number, %, optional). Complete JSON argument examples: {"control_visitors":1000,"control_conversions":50,"variant_visitors":1000,"variant_conversions":70} | {"control_visitors":5000,"control_conversions":200,"variant_visitors":5000,"variant_conversions":260} Outputs: control_rate_percent [%], variant_rate_percent [%], absolute_lift_percentage_points [pp], relative_lift_percent [%], standard_error_percentage_points [pp], z_score, p_value, significant, ci_lower_percentage_points [pp], ci_upper_percentage_points [pp], required_visitors_per_variant, verdict. Formula: p1 = x1/n1, p2 = x2/n2, p̄ = (x1 + x2)/(n1 + n2); z = (p2 − p1) / √(p̄(1 − p̄)(1/n1 + 1/n2)); p_value = 2·(1 − Φ(|z|)); CI = (p2 − p1) ± z_crit·√(p1(1 − p1)/n1 + p2(1 − p2)/n2); n per variant = (z_crit·√(2·p̄(1 − p̄)) + z_0.80·√(p1(1 − p1) + p2(1 − p2)))² / (p2 − p1)² Direct REST fallback: POST https://tttkmbb.com/api/v1/calculate/ab-test-significance with the same JSON input fields. Do not guess another /api/* path. Docs: https://tttkmbb.com/business/ab-test-significance.md