Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment, design an A/B test, calculate sample size, or interpret test results. Produces a complete test plan with hypothesis, variant definitions, sample size, duration estimate, guardrail metrics, and a results interpretation guide.
📚 Based on Controlled experiments — Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments
▸What is being tested — feature, UI change, copy, pricing, onboarding step
▸Hypothesis — or ask to help formulate one
▸Primary metric — conversion rate, click-through, completion rate, etc.
▸Baseline rate — and minimum detectable effect (MDE)
▸Daily eligible users — to calculate duration
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Hypothesis is directional (predicts a specific direction and magnitude, not "let's see")
✓Primary metric is singular (guardrail metrics are secondary)
✓Sample size is calculated from actual MDE and baseline (not guessed)
✓Test duration accounts for weekly seasonality (minimum 2 weeks)
✓Guardrail metrics are defined (at least one to protect revenue or core engagement)
✓Rollback trigger is specified with a concrete threshold
⚠️ What it refuses to do
Do not run a test without a directional hypothesis — "let's see what happens" produces uninterpretable results
Do not declare a winner before reaching the pre-planned sample size — peeking at results inflates false positive rates
Do not test multiple independent changes in a single variant — you won't know which change caused the result
Do not use engagement metrics (clicks, time-on-page) as the primary metric when the goal is revenue or retention — proxy metrics mislead
Do not ignore guardrail metrics — a conversion lift that causes a support ticket spike is not a win
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcp
Example output
Input: Test whether moving the signup CTA above the fold on the pricing page increases free-trial starts. Current trial-start rate 8%, ~20k weekly visitors. We want to detect a 10% relative lift. · generated by claude-sonnet-4-6