AE
AI Eval Plan
🔵 Stable🕐 updated 2026-06-24
🔷 SkillSpec L3
pm-ai
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
What to give it
▸The feature & task — what the model does and what "good output" means to a user.
▸Failure modes that matter — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow).
▸Available data — any real examples, logs, or labelled cases; or note there are none yet.
▸Who judges quality — automated checks, an LLM judge, human raters, or a mix.
▸The decision this gates — ship/no-ship, model selection, or prompt iteration.
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Each metric has an explicit rubric with anchors — not just a name
✓Deterministic/automated checks are used wherever possible before reaching for an LLM judge
✓The LLM judge is calibrated against human labels on at least a sample
✓The eval set includes adversarial and edge cases, not just happy-path examples
✓There is a single, explicit numeric bar for the ship decision
✓The plan specifies how it runs as a regression gate, not just a one-time check
⚠️ What it refuses to do
Do not rely on a single overall score — a feature can pass on average while failing every safety case
Do not trust an LLM judge you haven't calibrated against humans — it has its own blind spots and biases
Do not eval only on happy-path inputs — the failures live in the edges and the adversarial cases
Do not let the eval set leak into the prompt/few-shot examples — that's training on the test set
Do not define the pass bar after seeing the scores — set the threshold before you run, or it means nothing
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcpRelated skills
🔌 Embed this skill
Drop this on your blog, docs, or site — it renders a "Run this skill" card:
<div data-pm-skill="ai-eval-plan"></div>
<script src="https://mohitagw15856.github.io/pm-claude-skills/embed.js" async></script>💬 Discussion
AI Eval Plan is one of 551 open-source professional AI agent skills — all SkillSpec L3.
Try them all in the browser · ⭐ Star on GitHub · Browse the full catalog