AO
Agent Observability Spec
🔵 Stable🕐 updated 2026-07-01
🔷 SkillSpec L3
pm-agentops
Specify the tracing, metrics, and alerting for an AI agent or LLM feature in production. Use when asked what to log for an LLM app, design agent tracing or spans, define quality and cost monitors, or answer 'how do we know if the agent is misbehaving?'. Produces an observability spec with a trace schema, metric definitions with owners and alert thresholds, sampling and retention policy, and a privacy note for logged content.
What to give it
▸The system's shape — single LLM call, RAG pipeline, or multi-step tool-using agent
▸Traffic volume and cost sensitivity — full tracing at 10M req/day is a budget decision
▸What "misbehaving" means here — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
▸Existing observability stack — (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it
✅ The bar it holds itself to
Every skill in this library self-verifies — these are this skill's own quality checks, straight from its definition.
✓Any incident is replayable from its trace alone — the schema was tested against that bar
✓Every metric has a number, a window, and a named owner — no orphan dashboards
✓Quality alerts are drift-based against a rolling baseline, not absolute guesses
✓Sampling keeps 100% of error/guardrail/negative-signal traces
✓The privacy note exists and names retention and access — logged prompts are user data
⚠️ What it refuses to do
Do not log only inputs and outputs — without retrieval and tool spans, root cause analysis is guesswork
Do not alert on mean cost or mean latency — the tail is where both incidents live
Do not run judge-based quality scoring on 100% of traffic — sample; spend the budget on better baselines
Do not treat observability as launch-week scaffolding — drift metrics only work with months of baseline
Do not ship an agent that can take actions without logging the guardrail verdicts alongside the actions
Install
npx pm-claude-skills add --agent claude # or codex · cursor · gemini · hermes
# or one-line MCP (every skill, any client):
claude mcp add pm-skills -- npx -y pm-claude-skills-mcpRelated skills
🔌 Embed this skill
Drop this on your blog, docs, or site — it renders a "Run this skill" card:
<div data-pm-skill="agent-observability-spec"></div>
<script src="https://mohitagw15856.github.io/pm-claude-skills/embed.js" async></script>💬 Discussion
Agent Observability Spec is one of 551 open-source professional AI agent skills — all SkillSpec L3.
Try them all in the browser · ⭐ Star on GitHub · Browse the full catalog