A/B Test Setup

intermediatedataMin 16K context

Designs and implements A/B testing experiments including hypothesis formulation, sample size calculation, variant configuration, metric definition, and statistical analysis planning. Covers both frontend feature flags and backend experiment frameworks.

ab-testingexperimentationstatisticsproductanalyticsfeature-flags

Use Cases

  • Designing experiments with proper statistical rigor
  • Calculating required sample sizes for desired power
  • Setting up feature flag configurations for gradual rollouts
  • Defining primary and guardrail metrics for experiments
  • Analyzing experiment results and making ship/no-ship decisions

Example Prompt

Design an A/B test for the following change:

Feature: [describe the change]
Hypothesis: [what you expect to happen]
Primary metric: [what you're measuring]
Current baseline: [current metric value]
Minimum detectable effect: [smallest meaningful improvement]
Traffic: [daily active users or events]

Provide:
1. Experiment design (control vs variants)
2. Sample size calculation with 80% power, 95% confidence
3. Expected duration to reach significance
4. Guardrail metrics to monitor
5. Feature flag implementation code snippet
6. Analysis plan (when to check, how to decide)

Recommended Models

claude-opus-4-7gpt-5-5deepseek-v4

Compatible Tools

claude-codecursorkiroany

Modalities

Input: text
→
Output: text, code

Related Skills

data-analysis

Author

OpenModels Community

@openmodelsrun