PR
Prompt Regression Suite
🔵 Stable🕐 updated 2026-07-01
🔷 SkillSpec L3
pm-agentops
Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case pass criteria, CI gate thresholds, and a triage protocol for failures. For designing first-time evaluation of a new feature use ai-eval-plan instead.
What to give it
▸The feature and its contract — what the LLM step receives and must produce
▸What has broken before — (or nearly) — past incidents seed the best cases
▸Real traffic examples — 10-20 representative inputs, including ugly ones
▸What triggers a run — prompt edits, model bumps, retrieval changes, all of the above?
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Every past production failure appears as a permanent case
✓Canary cases cover the behaviours that must never change (refusals, format, safety)
✓No case relies on an LLM judge where a structural or property check would do
✓Gate thresholds are numbers, not "significant degradation"
✓The suite is fast and cheap enough that it actually runs on every change — state its runtime and cost
⚠️ What it refuses to do
Do not test only happy paths — the suite exists for the inputs that hurt you
Do not let anyone update golden expectations in the same PR that broke them, without review
Do not use an unpinned judge model — a judge that upgrades itself moves your baseline silently
Do not treat pass-rate-vs-baseline as the only gate — one dead canary matters more than 2% aggregate drift
Do not grow the set unboundedly — a suite too slow to run on every change protects nothing
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcpRelated skills
🔌 Embed this skill
Drop this on your blog, docs, or site — it renders a "Run this skill" card:
<div data-pm-skill="prompt-regression-suite"></div>
<script src="https://mohitagw15856.github.io/pm-claude-skills/embed.js" async></script>💬 Discussion
Prompt Regression Suite is one of 551 open-source professional AI agent skills — all SkillSpec L3.
Try them all in the browser · ⭐ Star on GitHub · Browse the full catalog