ER
Eval Rubric Designer
🔵 Stable🕐 updated 2026-06-29
🔷 SkillSpec L3
pm-ai
Design a scoring rubric and LLM-as-judge prompt to evaluate the quality of an AI feature's output. Use when asked to create an eval rubric, define quality dimensions, build an LLM judge, or decide how to measure whether AI output is good. Produces a rubric with weighted dimensions and concrete 1–5 anchors, a ready-to-run judge prompt, a labelling guide, and notes on judge reliability.
What to give it
▸The task — what the AI is supposed to produce, and for whom.
▸A sample output (or two) — ideally one good and one weak, to calibrate anchors.
▸What "good" means here — the quality bar and any non-negotiables (e.g. must be grounded, must follow format).
▸How it'll be scored — human review, LLM-as-judge, or both; and whether you need a single score or per-dimension.
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Dimensions are independent — a single flaw doesn't tank three of them at once
✓Every dimension has concrete 1/3/5 anchors specific to this task, not generic adjectives
✓The judge prompt has a strict, parseable output contract (JSON), with a retry/repair note
✓Weights reflect what actually matters for the task (grounding usually > prose polish)
✓The rubric is calibrated against at least one good and one weak sample
✓Judge biases are named with a concrete mitigation, not just listed
⚠️ What it refuses to do
Do not ship dimension names without anchors — names alone don't make scores reproducible
Do not let one quality issue load onto multiple dimensions — keep them orthogonal
Do not trust an LLM judge blind — calibrate against a handful of human labels first
Do not use a vague "overall quality 1–10" — it hides which part is broken
Do not ignore the negative case — a rubric must distinguish "wrong" from "thin", not just "great" from "okay"
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcpRelated skills
🔌 Embed this skill
Drop this on your blog, docs, or site — it renders a "Run this skill" card:
<div data-pm-skill="eval-rubric-designer"></div>
<script src="https://mohitagw15856.github.io/pm-claude-skills/embed.js" async></script>💬 Discussion
Eval Rubric Designer is one of 551 open-source professional AI agent skills — all SkillSpec L3.
Try them all in the browser · ⭐ Star on GitHub · Browse the full catalog