ED
Experiment Designer
🔵 Stable🕐 updated 2026-06-08
🔷 SkillSpec L3
pm-advanced
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. Produces a complete experiment design with hypothesis, sample size, run time, success criteria, and risk flags — or a results interpretation with ship/iterate/kill recommendation.
📚 Based on Pretotyping — Alberto Savoia, The Right It; Lean Startup (Eric Ries)
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Hypothesis specifies the change, the metric, the direction, and the reason
✓Primary metric is singular — guardrail metrics are secondary
✓Success criteria are defined before the test launches (not after seeing results)
✓Test was not stopped early (or flagged clearly if it was)
✓Practical significance assessed separately from statistical significance
✓Sample ratio mismatch is checked in results interpretation
⚠️ What it refuses to do
Do not define success criteria after seeing preliminary results — post-hoc success definitions are HARKing (Hypothesising After Results are Known) and invalidate the experiment
Do not stop a test early because the result looks significant — early stopping dramatically inflates false positive rates; the test must run to the planned sample size
Do not treat statistical significance as the same as practical significance — a p < 0.05 result with a 0.1% lift is real but may not be worth shipping
Do not run the same experiment on the same population multiple times without correction — multiple testing inflates the chance of a false positive proportionally
Do not use more than one primary metric — multiple primary metrics require multiple hypothesis corrections and make the ship/kill decision ambiguous
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcpRelated skills
🔌 Embed this skill
Drop this on your blog, docs, or site — it renders a "Run this skill" card:
<div data-pm-skill="experiment-designer"></div>
<script src="https://mohitagw15856.github.io/pm-claude-skills/embed.js" async></script>💬 Discussion
Experiment Designer is one of 1078 open-source professional AI agent skills — all SkillSpec L3.
Try them all in the browser · ⭐ Star on GitHub · Browse the full catalog